Several recent works (Liu et al., 2020; Soleimani et al., 2020; Zhao et al., 2020) leverage representations from large pre-trained language models (LMs) like BERT (Devlin et al., 2019), and RoBERTa  to achieve state-of-the-art results on FEVER.
it is unclear how factual knowledge encompassed in these LMs influences the verification process.