The way we construct regularization matrix may be lossy, risking losing information that is explicitly delivered in the lexicon.
it provides us effective encodings for words, and also yields better learning performance empirically in our experiments.