We did detect heads that paid increased (compared to the pre-trained BERT) attention to nouns and direct objects of the main predicates (on the MRPC, RTE and QQP tasks), and negation tokens (on the QNLI task), but the attention weights of such tokens were negligible compared to [CLS] and [SEP].
we believe that the striped attention maps generally come from BERT pretraining tasks rather than from task-specific linguistic reasoning.