In fact, Ostmeyer and Cowell 2017have presented a model that computes a recurrent weighted average (RWA) over every past hidden state.
the authors limit themselves to evaluate the model over simple tasks and the effectiveness of that model over language modelling is still to be demonstrated.