The dissimilarity is 0 if θ = φ, and increases as they diverge.
the KLdivergence between two word distributions conditioned on the same history u is We define D(φ, θ) to be the average KL-divergence over all histories: , which is convex in θ (Cover and Thomas, 1991).