As MLM is trained by masking only a part of the input sentence, replacing too many tokens can degrade its modeling performance.
we calculate the AUC until 20% of the tokens are replaced, and refer to it as AUC rep .