After obtaining the optimized θ * LM , we can use it to constrain the action space similar to the N-gram language model.
given previous t − 1 sampled words from current caption model, we compute p θ * LM (w t |w 0 , w 1 , ..., w t−1 ), which is the probability of the next word over the entire vocabulary.