Since one of the released BERT models was jointly pre-trained on 104 languages, 2 we use this multilingual BERT pre-trained model to encode the bilingual input pair of APE.
the whole encoder of our APE model is the multilingual BERT: we encode both src and mt in the same encoder and use the segment embeddings to differentiate between languages ( Figure 1).