The analyses have focused on wordlevel models, yet character-level models have been shown to outperform word-level models in some NLP tasks, such as text classification (Zhang et al., 2015), named entity recognition (Kuru et al., 2016), and time normalization (Laparra et al., 2018a).
there is a need to study pre-trained contextualized character embeddings, to see if they also yield improvements, and if so, to analyze where those benefits are coming from.