We note here that this procedure has three shortcomings: that we potentially bias the gold-standard towards (i) certain CoreSC labels through the use of our prioritisation table, (ii) the most reliable annotator and (iii) that the measure is conservative and favours singlelabel annotations.
in the case of three distinct labels the top three most popular labels are assigned scores of 0.6, 0.3 and 0.1 respectively.