In this work, we show that, on these tasks, polyglot character-based language models can provide benefits on top of those offered by contextualization.
even when crosslingual transfer with word type embeddings hurts target language performance relative to monolingual models, polyglot contextual representations can improve target language performance relative to monolingual versions, suggesting that polyglot language models tie dissimilar languages in an effective way.