Figure 4: Counter sample ratio for high vs low metric scored translations In order to shed some light on the reasons for the automatic evaluation failures when assigning low scores, we have manually analyzed cases in which a metric score is low but the quality according to humans is high (Q Nm ≤ 3 and Q h ≥ 7).
the main goal of our work is to analyze to what extent deep linguistic features can contribute to the automatic evaluation of translation quality.