With the obtained video frame features V = {f t } T t=1 and word embeddings of the sentence S = {w n } N n=1 , we employ two long shortterm memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997) to sequentially process the two different modalities, i.e., video and sentence, independently.
one LSTM sequentially models the video V , yielding the hidden states , while the other LSTM processes the sequential words in the sentence S, resulting in its corresponding hidden states {h s n } N n=1 .