Although plenty of text categorization datasets have been annotated, common practice reflects that initial noisy annotations be discarded and only consensus labels be published.
we follow previous work in achieving broad validation by constructing synthetic annotators that corrupt known gold standard labels.