2 The problem is that the fact of being easy or hard is not directly observable, but has to be inferred from the observed annotations.
some of the observed agreements will in fact be hard instances, since coin-flips could occasionally come out all-heads or all-tails.