In this part, we propose a sparse self-attention fine-tuning model (SSAF).
this finetuning model with BERT is composed of N sparse self-attention layers, where each layer learns a representation by taking the output from the previous layer:h where SSAM is adopted to replace the traditional self-attention mechanism, h 0 = embed(x) denotes the representation for the input sequence x which is the sum of token embeddings and the position embeddings, and LN is the layer normalization operation.