We find that on the same number of training instances (3360), the model performs better on cleaner data compared to singly-annotated data (r = 0.57 vs 0.64).
when we have a choice between collecting multiple annotations for the same instances vs collecting annotations for additional instances, the second strategy leads to more gains.