Compared to standard BERT's self-attention, ij , a K ij ∈ R dz are extra, which could be viewed as the edge representation between the input element x i and x j .
we devise a V ij and a K ij to encourage each token to be aware of the relative distance to different entity mentions, and vice versa.