Task Subjectivity In some tasks, aggregated annotations always correspond to the correct answer (Brew et al., 2010).
to fully utilize the crowd's knowledge, different approaches have been proposed to aggregate labels, from simply applying majority voting to more sophisticated strategies to assess annotators' reliability (Yang et al., 2018;Srinivasan and Chander, 2019;Rodrigues et al., 2014).