In this work, we present multi-granularity self-attention (MG-SA): a neural network that combines multi-head selfattention and phrase modeling.
we train several attention heads to attend to phrases in either n-gram or syntactic formalism.