In this experiment, our goal was to understand whether different syntactic and semantic relations are captured by self-attention patterns.
effects of this operation vary across tasks, and for QNLI and MNLI, it produces a performance drop of up to -0.2%.