Therefore, we explicitly add a regularization term to encourage the diversities between layers: where λ is a hyper-parameter and is set to 1.0 in this paper.
the regularization term measures the average of the distance between ev-ery two adjacent layers: Here D(H l , H l+1 ) is the averaged cosine-squared distance between the states in layers The cosine-squared distance between two vectors is maximized when two vectors are linearly independent and minimized when two vectors are linearly dependent, which satisfies our goal.