However, many agreement studies have restricted annotators to using a single sense, which can significantly lower inter-annotator agreement (IAA) in the presence of ambiguous or polysemous usages; indeed, multiple studies have shown that when allowed, annotators readily assign multiple senses to a single usage (Véronis, 1998;Murray and Green, 2004;Erk et al., 2009;Passonneau et al., 2012b).
we focus on annotation methodologies that enable workers to use as many labels as they feel appropriate, asking the question: if allowed to make labeling ambiguity explicit, will annotators agree?