Given the two distributions p and q above, the KL divergence is defined as The cross entropy and the KL divergence are related notions.
given the notations of L, p and q above, [5] shows that the cross entropy takes into account both the similarity between two distributions (given by KL divergence) and the entropy of the corpus in question, both of which contribute to the complexity of a LM task.