α t ∈ R k is the attentive weights of V and the attentive visual input z t ∈ R g is calculated as The visual input z t and the embedding of the previous output word y t−1 are the input of the LSTM.
there is a noticeable drawback that the previous output word y t−1 , which is a much stronger indicator than the previous hidden state h t−1 , is not used in the attention.