It is important to note that while the ELMo models we work with are monolingual, the BERT model is multilingual.
while the standalone ELMo models were trained on the tokenized WikiDump and CommonCrawl for each language respectively, the BERT model was trained only on the former, albeit simultaneously for 104 languages.