Traditionally, backing-off language models rely on the n-gram approximation, which are often criticized because they could only store limited information and thus lack any explicit representation of long range dependency.
recurrent neural network language models always estimate probabilities based on the full history (Sundermeyer et al., 2012) In other words, recurrent neural networks do not use limited size of context, which is the major reason why we choose recurrent neural network language models.