Then, we aggregate these scores in a vector z n ∈ R L and apply a transformation ρ to map them to a probability distribution α n ∈ ∆ L−1 (optionally taking into account the past cumulative attention β n−1 ): The "standard" choice for ρ is the softmax transformation.
in this work, we consider other possible transformations (to be described in §4).