The context of a word is often defined as the words appearing in a window of fixed-length (bag-of-words) and a simple approach is to treat the co-occurrence statistics of a word w as a vector representation for w (Mitchell and Lapata, 2008;Mitchell and Lapata, 2010); alternatively, dependencies between words can be used to define contexts (Goyal et al., 2013;Erk and Padó, 2008;Thater et al., 2010).
to distributional representations, NNLMs represent words in a low-dimensional vector space (Bengio et al., 2003;Collobert et al., 2011).