In this paper, we introduce a first study that investigates and compares different ways of aligning the pretrained contextualized embeddings.
we make the comparisons focused on the following properties: (1) aligning contextual-ized embeddings at the level of word tokens versus word types; (2) different training signals: static dictionaries, word alignment, or sentence alignment from parallel data; and (3) aligning different model variants: aligning from independently trained models versus aligning embeddings from a multilingual model with shared vocabulary.