Let p(x) and q(x) be two probability distributions of a random variable X.
the relative entropy of these distributions is defined as: D(p||q) is an asymmetric measure that considers the number of additional bits needed to encode p, when using an optimal code for q instead of an optimal code for p. : D(p||q) measures how much one probability distribution is different from another distribution.