The vocabulary of the words of a language encode information about what correct sequences of characters in a language look like.
we learn a language model over the character sequences in a vocabulary extracted from a large unlabeled corpus.