Owing to the large amounts of unlabeled data and the sufficiently deep architectures used during pre-training, advanced LMs such as BERT are able to capture complex linguistic phenomena, understanding language better than previously appreciated (Peters et al., 2018b;Goldberg, 2019).
as widely recognized, genuine reading comprehension requires not only language understanding, but also knowledge that supports sophisticated reasoning (Chen et al., 2016;Mihaylov and Frank, 2018;Bauer et al., 2018;Zhong et al., 2018).