However, as previous work (Huang et al., 1979;Jung et al., 2018) mentioned that the median value is more robust to outliers than the mean value, we use the median value to calculate the variance of the attention distribution.
local variance loss can be calculated as: where• is a median operator and is utilized to avoid zero in the denominator.