Levy and Goldberg (2014) make a connection between the NCE-based method of (Mikolov et al., 2013), and factorization of a matrix of pointwise mutual information (PMI) values of (x, y) pairs.
consistency of the NcE-based method under assumption 2.2 implies a similar result, specifically: if we define p N (y) = p Y (y), and define converge to values such that following (Levy and Goldberg, 2014), the inner product x is an estimate of the PMI up to a constant offset H(✓).