Recent advances in neural networks provide strong representational power to language models with distributed representations and unbounded dependencies based on recurrent networks (RNNs).
most language models operate by generating words by sampling from a closed vocabulary which is composed of the most frequent words in a corpus.