These results clearly show that the importance of the shortcut connections, and in particular, the semantic tasks in the higher layers strongly rely on the shortcut connections.
simply stacking the LSTM layers is not sufficient to handle a variety of NLP tasks in a single model.