A typical RNN-LM propagates a context vector that integrates information about previous inputs to use for the next prediction.
the information that is captured at the beginning of a sequence containing a long-distance dependency is likely to have faded from the context by the time the model spans that dependency.