In case of a large number of participating systems each assessor ranks only a subset of MT outputs.
a fair overall ranking cannot be always derived from such partial rankings (Callison-Burch et al., 2012).