In theory, we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment.
we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m n ⌧ 2 R M ⇥d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014;Weston et al., 2014).