Instead, we propose training the self-attention to attend to specific tokens corresponding to the syntactic structure of the sentence as a mechanism for passing linguistic knowledge to later layers.
we replace one attention head with the deep bi-affine model of Dozat and Manning (2017), trained to predict syntactic dependencies.