In Table 7, we further consider a scenario with 5 annotators, 4 Although there are 164 annotators in total in this dataset, each instance was labelled by only 10 of these annotators.
we use Krippendorff's alpha which is applicable to incomplete data to measure the inter-annotator agreement.