Previous research has shown the benefit of including various kinds of syntactic and semantic information into the LM (Khudanpur and Wu, 2000).
note that we can achieve 0.5% absolute WER improvement on the test set at only 1% of the standard MELM complexity.