To test this hypothesis we checked whether there are vertical stripe patterns corresponding to certain linguistically interpretable features, and to what extent such features are relevant for solving a given task.
we investigated attention to nouns, verbs, pronouns, subjects, objects, and negation words 6 , and special BERT tokens across the tasks.