Regarding contextual representation models, ELMo used the sampled softmax and GPT and BERT resorted to a subword method.
they used WordPiece (Wu et al., 2016) or BPE (Sennrich et al., 2016) to split the words into subwords and the language models were trained to take subwords as input and also predict subwords.