Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling complex functions and capturing complex linguistic structures.
nMT systems with deep architecture in their encoder or decoder Rnns often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often make the optimization much more difficult.