This asymmetry shows up in word-based models to a limited extent: In most models the unit of prediction is a word; predictors include n-grams of any size in principle, not just words.
in a class-based model the asymmetry between predictor and predicted is more important: There is no justification for the premise (made, for example, in the Brown model) that the classes that are optimal for predictors are also the classes that are optimal for predictees.