Then, under certain assumptions, given a large enough sample, the sample mean of the negative log probability of a model will converge to its cross entropy with the true model.
14 H(p,q) = -lira 1_ logq(w~) r/--+ Oo n where w~ is a string of the language L. In practice, one takes a large sample of the language, and calculates the negative log probability of the sample, normalized by its size.