Being Q h (t) and Q m (t) the human and metric assessed quality for the translation t, and being rank h (t) and rank m (t) the rank of the translation t according to humans and the metric, the normalized metric assessed quality is: In order to analyze the reliability of metrics when identifying wrong or high quality translations, we look for contradictory results between the metric and the assessments.
we look for metric errors in which the quality estimated by the metric is low (Q Nm (t) ≤ 3) but the quality assigned by assessors is high (Q h (t) ≥ 5) or viceversa (Q Nm (t) ≥ 7 and Q h (t) ≤ 4).