They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context.
such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context.