We posit that although annotator quality is an important factor in predicting accurate aggregations, the interplay between it and other factors is more nuanced.
rather than adapting the item-response method, our learning approach incorporates features that address multiple potential flaws or biases in crowdsourced annotations.