Another line of work has approached the problem of aggregating labels from multiple annotators (Raykar et al., 2009;Hovy et al., 2013;Yan et al., 2013).
if we believe that some annotators are more reliable than others, it might make sense to try to determine this in an unsupervised manner, and give more weight to the annotations from the reliable annotators.