In this paper, we develop a novel Sparse Self-Attention Fine-tuning model (referred as SSAF) which integrates sparsity into selfattention mechanism to enhance the finetuning performance of BERT.
sparsity is introduced into the self-attention by replacing softmax function with a controllable sparse transformation when fine-tuning with BERT.