Similarly, LMs encode a specific context as the hidden states of RNNs, and use them to predict future contexts.
we recognize the cell states of LSTMs are more like to be the sentence embedding (Radford et al., 2017), since they are only passed to the next time stamps.