Moreover, as Transformers are fundamental components of BERT, multi-head self-attention layer (Vaswani et al., 2017) is used to extract useful information within the input sequence.
in our model, to enhance the target representation, we use a multi-head self-attention layer to extract useful information in source BERT representation B S .