Note that we use a slightly updated residual structure as implemented by tensor2tensor 1 than proposed originally.
with recent advances in network architectures for Neural Machine Translation (NMT) recurrent models have effectively been replaced by either convolutional or self-attentional approaches, such as in the Transformer.