(2017) have also shown that while LSTMs are aware of which words appear in their context, this awareness degrades with increasing length of the sequence.
the success of copy mechanisms such as attention and caching (Bahdanau et al., 2015;Hill et al., 2016;Merity et al., 2017;Grave et al., 2017a,b) suggests that information in the distant context is very useful.