Furthermore, annotators often disagree on what constitutes a mistake, and some gold errors missed by a system may be considered as acceptable usage by another rater.
when a system is compared against the gold truth produced by just one annotator, performance is understated.