The original BERT model is pre-trained with both Wikipedia and the Books corpus (Zhu et al., 2015), the former of which contains our (unrevised) test targets.
we also pre-train a version of BERT on the Books corpus only, which we consider a more correct baseline.