Neural machine translation (NMT) typically makes use of a recurrent neural network (RNN) -based encoder and decoder, along with an attention mechanism (Bahdanau et al., 2015;Cho et al., 2014;Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014).
it has been shown that RNNs require some supervision to learn syntax (Bentivogli et al., 2016;Linzen et al., 2016;Shi et al., 2016).