semantic similarity, paraphrasing, entailment).
for de-ATTN-CTX models, we observed a much  flatter distribution of attention weights for each head and, unlike in the other models, we were often able to identify a head focusing on the main verb.