To get a better understanding of how well human annotators perform at the task, we compute the one-vs-rest Pearson correlation coefficient using the gold standard annotations.
for each topic, we single out each rating/annotation and compare it to the average of all other ratings/annotations.