In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads.
we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads.