To overcome the above limitation, in this work, we develop a novel Sparse Self-Attention Finetuning model (referred as SSAF) by integrating sparsity into self-attention to refine the attention distribution when fine-tuning the pretrained BERT.
we propose sparse self-attention mechanism (SSAM) to induce sparsity by replacing the traditional softmax function with a sparse transformation in self-attention networks.