Multiple-tasks supervision at the outermost layer has a strong tradi-tion in neural net models in vision and elsewhere (Caruana, 1993;Zhang and Zhang, 2014;Yim et al., 2015).
this suggests having task supervision for low-level tasks at the lower bi-RNN layers.