The same conclusion can also be drawn from the success of a concurrent work ELMo which uses LM representations as word vectors and achieves state-ofthe-art results on six language tasks (Peters et al., 2018).
if we increase the context length to three sentences, the accuracy of 1B-LM only has a marginal improvement.