Thus, MI(X, Y ) is the degree of dependence between X and Y measured by the KL divergence between P XY and P X P Y .
here, by definition of the KL divergence, MI can be represented in the form of the expectation over P XY , i.e., the summation over all possible pairs (x, y) ∈ X ×Y: The shaded part in Equation 5is actually the pointwise mutual information (PMI) (Church and hanks, 1989): PMI(x, y) can be thought of as the contribution of (x, y) to MI(X, Y ).