They showed that a machine-learning classifier is sensitive to the type of noise in the data.
if the noise is in the form of category over-use (an annotator disproportionately favors a certain category), when algorithm performance is measured against the noisy data, accuracy estimates are often inflated relative to performance on the real data, uncorrupted by noise (see Figure 3(b) therein).