Although the first one has proven to be beneficial during pre-training to some NLP tasks, we want to check how much its influence is to our final translation performance.
we disable those two modifications in CMLM one by one and report the translation results.