We also compute the κ values for each pair of annotators, and average them for each annotator (similar to the approach in Tratz and Hovy (2010)).
whenever one label is more prevalent (a common case in NLP tasks), κ overestimates the effect of chance agreement (Feinstein and Cicchetti, 1990) and penalizes disproportionately.