Frankle and Carbin (2018) showed that widely used complex architectures suffer from overparameterization, and can be significantly reduced in size without a loss in performance.
it is important to note that across all tasks and datasets, disabling some heads leads to an increase in performance.