ELMo uses a character-CNN but it is relatively slow.
we use a pre-trained open-vocabulary word embedding as the input layer of the contextual encoder, reducing the time complexity of the input layer to a negligible level.