The interpretable attention can better explain how the model processes the natural language and guide the future improvement.
giving both s i,i+1 and s i,i−1 large values, so the attention head freely attends to any position without restriction of constituent prior, which is the same as the original Transformer.