We present the first empirical analysis of Transformers with sparse attention mappings (i.e., entmax), showing potential in both translation accuracy as well as in model interpretability.
we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence.