Previous studies have shown the superiority of this architecture on both language modeling (Pascanu et al., 2014) and machine translation (Barone et al., 2017; Meng and Zhang, 2019).
meng and Zhang (2019) propose to maintain a linear transformation path throughout the deep transition procedure with a linear gate to enhance the transition structure.