The idea of the modality-based attention is to prioritize one of the modalities at the word level.
when the lexical features are more relevant to capture emotions (i.e., informative words are used), the model should prioritize such features and vice versa (i.e., arousal and pitch levels increase).