We argue other abilities of non-recurrent networks could be responsible for their strong performance.
we hypothesize that the improvements in BLEU are due to CNNs and Transformers being strong semantic feature extractors.