We selected the ReLU activation function except for the attention layers.
the modality fusion outperforms both shallow fusion (directly using SVM classifier) and deep fusion (DF) on diverse feature extraction combinations.