(Lu et al., 2016) proposed a "visual sentinel", which applies the hidden state in LSTM as supervision to adaptively decide if it is necessary to input visual feature to the language model when generating the next word.
the supervision information from the hidden state isn't credible which cannot make sure it contains the corresponding visual decision signals.