Figure 1a shows that the difference in dev loss, between truncated-and infinite-context variants of the test setting, gradually diminishes as we increase n from 5 tokens to 1000 tokens.
we only see a 1% increase in perplexity as we move beyond a context of 150 tokens on PTB and 250 tokens on Wiki.