It employed 50 independent annotators for a "graded" textual inference task, yielding a total of roughly 19,840 annotations, and validates that disagreements among the annotations are reproducible signals.
in their work, the labeling schema is modified from 3-way categorical NLI to a graded one, whereas our study keeps the original 3-way labeling schema to facilitate a direct comparison between old labels and new labels, and focuses more on giving an in-depth analysis regarding the relation between the level of disagreements among humans and the state-of-the-art model performance.