Thus, in order to interface with existing NLI models, we discretize 10 our collected human judgments by mapping the raw (unnormalized) score (which is between −50 and 50) into K evenly sized bins, where K is equal to the number of classes that were used in the original dataset from which the p/h pair was drawn.
for pairs drawn from datasets which use the three-way ENTAILMENT/ CONTRADICTION/NEUTRAL labels (i.e., SNLI, MNLI, and RTE2), we consider human scores less than −16.7 to be CONTRADICTION, those greater than 16.7 to be ENTAILMENT, and those in between to be NEUTRAL.