The purpose of automatic aggregation models such as those decribed in Section 2 is to arrive at the same judgment with a few annotations that a simpler scheme like majority vote would have arrived at given many annotations.
a crowd-constructed gold standard is appropriate for evaluation of such models.