To judge the quality of a machine translation, one measures its closeness to one or more reference human translations according to a numerical metric.
our MT evaluation system requires two ingredients: 1. a numerical "translation closeness" metric 2. a corpus of good quality human reference translations We fashion our closeness metric after the highly successful word error rate metric used by the speech recognition community, appropriately modified for multiple reference translations and allowing for legitimate differences in word choice and word order.