As seen in the previous two experiments, EASL achieves higher Spearmans correlation on ranking MT systems with smaller number of annotations than the baseline method (DA), 21 This is the same setting as WMT14, WMT15, and WMT16 (Bojar et al., 2014(Bojar et al., , 2015, although they used TrueSkill (Gaussian) instead of EASL to rank systems.
direct assessment or direct annotation (dA) is a straightforward method for collecting graded response from annotators.