the ones that could be informative) varies from 32% (MRPC) to 61% (QQP) depending on the task.
manual inspection of self-attention maps for both basic pre-trained and fine-tuned BERT models suggested that there is a limited set of selfattention map types that are repeatedly encoded across different heads.