Assuming a particular video has 'u' utterances, the raw utterance level multi-modal features are rep- Three separate Bi-GRU layers with forward & backward state concatenation are first applied on the raw data followed by the fully-connected dense layers, resulting in T ∈ R u×d (text), V ∈ R u×d (visual) and A ∈ R u×d (acoustic), where 'd' is the number of neurons in the dense layer.
finally, pairwise-attentions are computed on various combinations of three modalities- the attention between V and T is computed as follows: • Bi-modal Attention: Modality representations of V & T are obtained from the Bi-GRU network, and hence contain the contextual information of the utterances for each modality.