In addition to human action recognition, our work relates to other multimodal tasks such as visual question answering (Jang et al., 2017;Wu et al., 2017), video summarization (Gygli et al., 2014;Song et al., 2015), and mapping text descriptions to video content (Karpathy and Fei-Fei, 2015;Rohrbach et al., 2016).
we use an architecture similar to (Jang et al., 2017), where an LSTM (Hochreiter and Schmidhuber, 1997) is used together with frame-level visual features such as Inception (Szegedy et al., 2016), and sequence-level features such as C3D (Tran et al., 2015).