Additionally, the 1B-LM is trained on the sentence level, which might also result in the inability to track paragraph level information.
to investigate the differences between training on sentence level and on paragraph level, a prohibitive amount of computational resource is required to train a large model on the 1 Billion Word Corpus.