We explore neural encoder-decoder models based on Transformer Networks (Vaswani et al., 2017).
to RNNs, Transformers abandon recurrence in favor of a mix of different types of feedforward layers, e.g., in the case of the Transformer decoder, self-attention layers, cross-attention layers (attending to the encoder outputs), and fully connected feed-forward layers.