Chance-Adjusted Agreement: Agreement coefficients measure the proportion of observed agreements that are above the proportion expected by chance.
given an estimate A m,n of the probability that two annotators m, n ∈ 1:J will agree on a label and an estimate of the probability C m,n that they will agree by chance, chance-adjusted agree- Chance agreement takes into account the prevalence of the individual labels in 1:K. it is defined to be the probability that a pair of labels drawn at random for two annotators will agree.