With more gains arising from continued research on new neural network architectures and accompanying train-ing techniques (Vaswani et al., 2017;Gehring et al., 2017;Chen et al., 2018), NMT researchers, both in industry and academia, have doubled down on their ability to train high capacity models on large corpora with gradient based optimization.
despite huge improvements in overall translation quality NMT has shown some glaring weaknesses, including idiom processing, and rare word or phrase translation (Koehn and Knowles, 2017;Isabelle et al., 2017;Lee et al., 2018) tasks that should be easy if the model could retain learned information from individual training examples.