However, although it is standard practice in MT evaluation to measure increases in automatic metric scores with significance tests (Germann, 2003;Och, 2003;Kumar and Byrne, 2004;Koehn, 2004;Riezler and Maxwell, 2005;Graham et al., 2014), this has not been the case in papers proposing new metrics.
it is possible that some reported improvements in correlation with human judgment are attributable to chance rather than a systematic improvement.