The Kullback-Leibler (KL) divergence function is a distributional similarity function that is of particular relevance here since it can be described informally in terms of substitutability.
given cooccurrence distributions p and q, its mathematical definition can be written as:  (e) w1 and w2 are EXCLUSIVE Figure 3: Surprise in substituting w 2 for w 1 (darker shading indicates higher surprise) The value log 1 p(x) has an informal interpretation as a measure of how surprised an observer would be to see event x, given prior likelihood expectations defined by p. if p and q are the distributions of words w 1 and w 2 then where E p is the expectation function over the distribution of w 1 (i.e.