We use masked language model probabilities computed by pretrained BERT model 2 as an additional feature during rescoring to further improve our sentence-level baseline and our cross-sentence GEC model.
we replace tokens in a hypothesis T with the [MASK] token, one at a time, predicting the log probability of the masked token using BERT.