Adding noise to input sentences has been very benefi-cial for the autoencoder setups of (Lample et al., 2018a;Hill et al., 2016) which is inspired by denoising autoencoders (Vincent et al., 2008).
we transform source sentences with three types of noise: deleting words with probability 0.1, replacing words by a filler token with probability 0.1, and swapping words which is implemented as a random permutation over the tokens, drawn from the uniform distribution but restricted to swapping words no further than three positions apart.