Under these conditions, setting different priors to reflect the annotator pool should improve performance.
whenever one label is more prevalent (a common case in NLP tasks), κ overestimates the effect of chance agreement (Feinstein and Cicchetti, 1990) and penalizes disproportionately.