We would like as much context as possible, but wordsequence contexts become less probable as their length increases, making learning harder.
simple models of context are preferred, although the precise definition varies between systems.