(2016) that proposed a visual sentinel gate to make a decision on whether the model should attend to the image or to the sentinel gate, an additional gating cell is introduced on top of the traditional LSTM to gate another controlling vector s. Figure 6 shows how RAL-STM controls the DA vector s. First, starting from the 1-hot vector of the DA s 0 , at each time step t the proposed cell computes how much the LSTM outputh t affects the DA vector, which is computed as follows: where W ax , W ah are weight matrices to be learned.
the new input x t consists of information of the original input token w t , the DA representation d t , and the hidden context h t 1 .