Second, many studies using crowdsourcing combine the results into a single answer, thereby leveraging the wisdom of the crowds (Surowiecki, 2005) to smooth over inconsistencies in the data.
in the second experiment, we evaluate different methods of combining Turker responses into a single sense labeling, referred to as an aggregate labeling, and comparing that with the reference labeling.