To alleviate this issue, we adopt the denoising auto-encoder (DAE) following the previous work (Lample et al., 2019) that tries to reconstruct the corrupted input.
we randomly shuffle a small portion of words in the original sentence to the corrupted input which are then fed to the denoising auto-encoder.