The GloVe model is closely related to the notion of pointwise mutual information (PMI), which is defined for two words i and j as PMI(i, j) = log P (i,j) P (i)P (j) , where P (i, j) is the probability of seeing the words i and j if we randomly pick a word position from the corpus and a second word position within distance W from the first position.
the PMI between i and j is usually estimated as follows: where it is straightforward to see that after the reparameterization given by (1) In this paper, we will use the following variant of the formulation in (1): (2) Despite its similarity, this formulation differs from the GloVe model in a number of important ways.