This matches our intuition: without generalizing to semantic language models, higher order language models will be relatively sparse and contain much noise.
when taking into account the semantic features, we found that bigram and trigram semantic language model fea-tures outperformed unigrams.