To resolve this issue, Transformer-XL  introduces the idea of recurrence to the transformer language model.
the modeling of a new language segment in Transformer-XL is conditioned on hidden states from previous language segments.