(2013) to compare evaluation consistency.
using system-level data from the WMT16 Metrics Task, we computed the average difference between the scores assigned to the same MT output by different judges and by the same judge.