At the same time we modify our surface realization component so that it takes note of the image annotation probabilities.
we use an adaptive language model (Kneser et al., 1997) that modifies an n-gram model with local unigram probabilities: •P(len(C) = n) where P(w i ∈ C|w i ∈ D) is the probability of w i appearing in the caption given that it appears in the document D, and P adap (w i |w i−1 , w i−2 ) the language model adapted with probabilities from our image annotation model: where P back (w|h) is the probability of w given the history h of preceding words (i.e., the original trigram model), P adap (w) the probability of w according to the image annotation model, P back (w) the probability of w according to the original model, and β a scaling parameter.