This is illustrated by Figure 7 where without caching, C near words see a 22% increase in perplexity for PTB, and a 32% increase for Wiki, whereas C far see a 28% increase in perplexity for PTB, and a whopping 53% increase for Wiki.
the cache is, in a sense, complementary to the standard model, since it especially helps regenerate words from the long-range context where the latter falls short.