This produces dramatic improvements over a range of NLP tasks where appropriate unlabelled data is available (Peters et al., 2017(Peters et al., , 2018Akbik et al., 2018;Devlin et al., 2019).
there is still a lack of systematic study on how to select appropriate data to pretrain word vectors or LMs.