However, we also wish to investigate how well individual annotators perform.
we also compute error and pairwise correlation for each individual annotator with the authors or other annotators, and then average over the pairwise comparisons for each annotator type.