The second is that the evaluations have poor reproducibility due to the difficulty of reaching agreement on the scoring criteria among the evaluators.
in the past decade, evaluation methods that automatically evaluate MT quality based on its similarity to a reference translation have become common (Papineni et al., 2002;Doddington, 2002;Banerjee and Lavie, 2005;Snover, et al., 2006;Melamed et al., 2007;Isozaki et al., 2010;Birch and Osborne, 2011).