The output of the group attention O is produced by: Following the same paradigm in (Vaswani et al., 2017), we add a fully-connected feed forward layer to the multi-head attention mechanism layer (i.e., group attention), and each layer is followed by a residual connection and layer normalization.
the output of group attention block O is obtained.