Hence, ranking-based evaluations, where judges are asked to rank the output of 2 to 5 systems, have been used in recent years, which has yielded much higher inter-annotator agreement (Callison-Burch et al., 2007).
finally, we have presented evidence suggesting that using the pairwise hidden layers is advantageous over simpler flat models.