To better understand the difficulty of the task, we compute the agreement between human annotators by calculating the Pearson correlation coefficient of model precisions produced by randomised sub-group pairs in the topics.
8 for each topic, we randomly split the annotations into two sub-groups, and compute the Pearson correlation coefficient of the model precisions produced by the first sub-group and that of the second subgroup.