(2018), where an attention head is replaced by a model trained to predict syntac-tic dependencies (Dozat and Manning, 2017).
our model uses explicit supervision for all self-attention heads and is trained to predict the correct attention scores in a multi-task fashion.