The goal of this study is to establish whether such disagreements can safely be attributed to ''noise'' in the annotation process (resolvable via aggregation), or rather are a reproducible signal and thus should be treated as part of the NLI label assigned to the p/h pair.
our primary contributions are: • We perform a large-scale study of humans' sentence-level inferences and measure the degree to which observed disagreements persist across samples of annotators.