The distribution of attention is then regularized with the learned Gaussian bias to produce the final distribution, which pays more attention to the local context around the word "talk".
a Gaussian bias G is placed to mask the logit similarity energy in Equation 2, namely: The first term is the original dot product selfattention model.