Vilnis and McCallum (2014) recently proposed an alternative view, where words are represented by a whole probability distribution instead of a deterministic point vector.
they model each word by a Gaussian distribution, and learn its mean and covariance matrix from data.