• We empirically show that the number of attention heads in Transformers impacts their ability to capture long-distance dependencies.
many-headed multi-head attention is essential for modeling longdistance phenomena with only self-attention.