To further complicate matters, some workers may produce spam or malicious responses.
we posit that although annotator quality is an important factor in predicting accurate aggregations, the interplay between it and other factors is more nuanced.