We have shown that models that are trained to predict an aggregate score do not, by default, model the same type of uncertainty as that which is captured by distributions over many human raters' judgments.
several challenges would need to be overcome to switch to the proposed NLI evaluation.