Despite the fact that we reset the memory for the Attentive RNN-LM at each sentence boundary whereas the caches for the baseline systems span sentence boundaries, our best Attentive RNN-LM is within 1 perplexity point of the system of (2017) (which is allowed to cache 2,000 previous hidden states), and has a lower perplexity than all of the other baselines.
the words in the X-axis (horizontal) are the inputs at each timestep and the words in the Y-axis (vertical) are the next (or predicted) words.