Appendix A.2): For using the sparsemax mapping in an attention mechanism, Martins and Astudillo 2016show that it is differentiable almost everywhere, with where At first glance, sparsemax appears very different from softmax, and a strategy for producing other sparse probability mappings is not obvious.
the connection becomes clear when considering the variational form of softmax (Wainwright and Jordan, 2008): where H S (p) := − j p j log p j is the well-known Gibbs-Boltzmann-Shannon entropy with base e. Likewise, letting H G (p) := 1 2 j p j (1 − p j ) be the Gini entropy, we can rearrange Eq.