We take care never to allow backoff to the empty state , 2 since we find that c(w) is otherwise too coarse an approximation to English: sampled sentences tend to be disjointed, with some words generated in complete ignorance of their left context.
the "variational gap" between (4) and 5is the bound is good if q does a good job of approximating p θ 's tagging distribution on a randomly drawn sentence.