Another idea is to replace the softmax in soft attention with sparsity inducing operators (Martins and Astudillo, 2016; Niculae and Blondel, 2017).
all sparse/local attention methods continue to compute P (y) from an attention weighted sum of inputs (Eq: 2) unlike hard attention.