For this task, the agreement between the attention weights of the models and humans (using majority voting) is lower than for the previous task.
the IAA between Turkers is also just moderate (α between 0.66 and 0.72), which shows that this is inherently a difficult task.