For example, in a binary task with 95% prevalence of one category, two 90% accurate annotators would have negative chanceadjusted agreements of 0.9−(.95 2 +.05 2 ) 1−(.95 2 +.05 2 ) = −.053.
high chance-adjusted interannotator agreement is not a necessary condition for a high-quality corpus.