In theory, RNNs like long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and GRU should propagate long-term contextual information.
in practice it is not always the case (Bradbury et al., 2017).