Stolcke used the Kullback-Leibler divergence between the pruned and un-pruned model probability distribution in a given context over the entire vocabulary.
the increase in relative entropy from pruning a bigram is computed by where the summation is over all word pairs (w i-1 w i ).