Their semantic tasks also improve when introducing shortcut connections, i.e., feeding the word representations into all layers of their network.
alonso and Plank (2017) find generally mixed performance of MTL for semantic tasks.