Unsupervised classbased language models such as Random Forest LM (Xu and Jelinek, 2007), Model M (Chen, 2008) have been investigated that outperform a word-based LM.
the long-distance information is captured by using a cache-based LM that takes advantage of the fact that a word observed earlier in a document could occur again.