The gap is larger when the LM is trained on the 1 Billion Word Corpus, indicating that more training data results in a better generalization.
the accuracy of 1B-LM is 0.695 when one sentence is used as the context.