Prior work has shown that existing automatic metrics have poor instance-level correlation with mean human judgment and that they score many good quality responses poorly.
the evaluation is systematically biased against genuine system improvements that would lead to higher human evaluation scores but not improve automatic metrics.