In all of our experiments, we mask 15% of all WordPiece tokens in each sequence at random.
to denoising auto-encoders (Vincent et al., 2008), we only predict the masked words rather than reconstructing the entire input.