Theorem 2: The difference between the SGNS score at the PMI matrix and the SGNS score at a given matrix m can be written as: The KL divergence between two distributions is always non-negative and is zero only if the two distributions are the same.
we rederive the results of (Levy and Goldberg, 2014) that S(pmi) = max m S(m).