The objective that SGNS minimizes is In the limit of large d, Levy and Goldberg (2014) found the following closed-form solution to Eq 1: The first term can be seen as an empirical estimate of Pointwise Mutual Information (PMI): PMI(w, c) = log P (w,c) P (w)P (c) .
the matrix M that SGNS factorizes can be constructed from For computational convenience, Levy and Goldberg (2014) suggested a sparse and consistent alternative called Shifted Positive PMI (SPPMI): (2) Levy and Goldberg (2014) showed that using such sparse representation, word and context embeddings could be efficiently obtained using a truncated singular value decomposition.