In either case, a primary benefit of contextualized word embeddings is that they seed the learner with distributional information from large unlabeled datasets.
the texts used to build pretrained contextualized word embedding models are drawn from a narrow set of domains: • Wikipedia in BERT (Devlin et al., 2019) and ULMFiT (Howard and Ruder, 2018); • Newstext (Chelba et al., 2013) in ELMo (Peters et al., 2018); • BooksCorpus (Zhu et al., 2015) in BERT (Devlin et al., 2019) and GPT (Radford et al., 2018).