An LM provides a probability for a sequence of words in a given language, reflecting fluency and the likelihood of that word sequence occurring in that language.
we see our "attentive" RNN-LM (see §3) as a generalized version of these models as we rely on the encoded information in the hidden state of the RNN-LM to represent previous input words and we use a set of attention weights (instead of a key) to retrieve information from the past inputs.