Instead of taking the mean of the intermediate recurrent layer states h t , we could use the last state vector h M to compute the score and remove the mean-over-time layer.
as we will show in Section 5.2, it is much more effective to use the mean-over-time layer and take all recurrent states into account.