Very few works try to interpret recurrent neural networks in NLP.
an improvement over the lower bound models indicates that the encoder learns syntactic information, whereas a decline from the upper bound model shows that the encoder loses certain syntactic information.