Even if NLI is treated as a gradable phenomenon (as we believe it should be), a world in which all humans share the same notion of uncertainty necessitates very different models, annotation practices, and modes of evaluation than a world in which people may disagree substantially in specific situations, use different heuristics, and/or have different preferences about how to resolve uncertainty.
current practicesin which we aggregate human judgments through majority vote/averaging and evaluate models on their ability to predict this aggregated label-are only appropriate if humans all tend to use the same process for resolving uncertainties in practice.