Levy and Golberg (2014c) show that SGNS's corpuslevel objective achieves its optimal value when: Hence, SGNS is implicitly factorizing a wordcontext matrix whose cell's values are PMI, shifted by a global constant (log k): SGNS performs a different kind of factorization from traditional SVD (see 2.2).
the factorization's loss function is not based on L 2 , and is much less sensitive to extreme and infinite values due to a sigmoid function surrounding w • c. Furthermore, the loss is weighted, causing rare (w, c) pairs to affect the objective much less than frequent ones.