The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform 'zeroresource' cross-lingual classification on the XNLI dataset (Conneau et al., 2018).
when multilingual BERT is finetuned for XNLI with English data alone, the model also gains the ability to handle the same task in other languages.