Each output element z i is a weighted sum of a linear transformation of input elements: where in Equation 2 are obtained by the self-attention model, which captures the correspondences between x i and others.
the attention weight α ij of each element x j is computed using a softmax function: where is an alignment function which measures how well the input elements x i and x j match.