This self-attention is performed for each token for each head, and the results of the H self-attentions are concatenated to form the final self-attended representation for each token.
consider the matrix S (j 1) of T token representations at layer j 1.