Surprisingly, for exponential language models, the number of model parameters seems to matter not at all; all that matters are the magnitudes of the parameter values.
one can improve such models by adding features (or a prior model) that reduce parameter values while maintaining training performance.