Given two random variables X and Y , a conditional probability mass function , and a marginal probability mass function , the conditional entropy of From Equation (3.8), we can see that the first three items are the values computed by the normal word trigram model as shown in Equation (2.9) and the forth item contributes to summation of the mutual information of the next word with the words in the history .
we call Equation (3.8) as a MI-Ngram model and rewrite it as: For a large enough corpus, the conditional perplexity is usually an indication of the amount of information conveyed by the model: the lower the conditional perplexity, the more information it conveys and thus a better model.