While the unmasked model achieves a 83.6 F 1 on the original SemEval dev set, F 1 drops drastically to 62.4 if we replace dev set entity mentions with a special <UNK> token to simulate the presence of unseen entities.
the masked model is unaffected by unseen entity mentions and achieves a stable dev F 1 of 74.7.