In recent years, there have been an increasing number of studies (Su et al., 2007;Kittur et al., 2008;Sheng et al., 2008;Snow et al., 2008;Callison-Burch, 2009) using crowdsourcing for data annotation.
because annotators that are recruited this way may lack expertise and motivation, the annotations tend to be more noisy and unreliable, which significantly reduces the performance of the classification model.