(2019) extended this work to using multiple layers and tasks, supporting the claim that BERT's intermediate layers capture rich linguistic information.
the gain from disabling a single head is different for different tasks, ranging from the minimum absolute gain of 0.1% for StS-B, to the maximum of 1.2% for MRPC (see Figure 8).