Concretely, we replace the row normalization π in Equation 1 by This change leads to sparse attention weights, as long as α > 1; in particular, α = 1.5 is a sensible starting point (Peters et al., 2019).
this is the first work to allow such atten-  -entmax Figure 9: Interrogation-detecting heads in the three models.