We also hypothesize that the attention distribution in Trans-noEnc is not spread out enough for extract-ing contextual features.
we find that word embeddings and attention distributions are not the major reasons causing the distinct gap.