Although BERT was pretrained on a corpus comprising 3.3B words , it is reasonable to assume that it is easier to procure abundant monolingual data than parallel data.
we empirically verified that a simplistic context average baseline outperforms contextualized word embeddings in a variant of our task which relies on parallel contexts.