To further guarantee the diversity, we enlarge the distances among multiple attention heads with disagreement regularization (Section 3.1).
we propose three types of disagreement regularization to encourage each head vector O i to be different from other heads (Section 3.2).