Similarly, upper layers of LSTMs trained on NLP tasks learn more task-specific representations (Liu et al., 2019a).
it follows that upper layers of neural language models learn more context-specific representations, so as to predict the next word for a given context more accurately.