For the three modalities, three separate attention blocks are required, where each block takes multi-utterance information of a single modality and computes the self attention matrix.
the MU-SA attention (A v ) on V (visual) will be computed as follows, Mv(i,j) u k=1 e Mv(i,k) for i, j = 1, .., u The attention matrix A p ∈ R 3×d is computed for p = 1 st , 2 nd , ...u th utterances.