The authors of BERT recommend not to mask words randomly with [MASK] when fine-tuning the network.
we discovered that masking often reduces the tendency of the classifiers to overfit to BERT by forcing the network to rely on the context of surrounding words.