In crowdsourcing, each instance is presented to multiple expert or non-expert annotators for labelling.
labels collected in this manner could be noisy, since some annotators could produce a significant number of incorrect labels.