Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text.
the applicability of this approach is unknown when the target domain varies substantially from the pretraining corpus.