Our proposed alternative is to calibrate the reference systems against an existing set of human judgments for a range of outputs from different MT systems.
if many of the reference system's outputs are similar to those MT outputs that received low assessments, we conclude this reference system may not be of high quality.