Probabilistic methods for aggregating crowdsourced data have been shown to be more accurate than simple heuristics such as majority vot-ing (Raykar et al., 2010;Sheshadri and Lease, 2013;Rodrigues et al., 2013;Hovy et al., 2013).
existing methods for aggregating sequence labels cannot model dependencies between the annotators' labels (Rodrigues et al., 2014;Nguyen et al., 2017) and hence do not account for their effect on annotator noise and bias.