This is contrary to other results in the machine translation community (e.g., Vaswani et al., 2017), but could be explained by the fact that we are not using any learning rate schedule for training Transformers.
they never manage to outperform Transformers in terms of test BLEU score of the final model.