However, Wallach's experiments were limited to bigrams, a toy setting for language models, and experiments with higher-order n-grams have not yet been sufficiently studied, which we investigate in this paper.
we point out the two fundamental problems caused when extending Wallach's model to a higher-order: sparseness caused by dividing all n-grams into exclusive topics, and local minima caused by the deep hierarchy of the model.