First, the choice in sense inventory plays an important role in gathering high-quality annotations; fine-grained inventories such as WordNet often contain several related senses for polysemous words, which untrained annotators find difficult to correctly apply in a given context (Chugur et al., 2002;McCarthy, 2006;Palmer et al., 2007;Rumshisky and Batiukova, 2008;Brown et al., 2010).
many agreement studies have restricted annotators to using a single sense, which can significantly lower inter-annotator agreement (IAA) in the presence of ambiguous or polysemous usages; indeed, multiple studies have shown that when allowed, annotators readily assign multiple senses to a single usage (Véronis, 1998;Murray and Green, 2004;Erk et al., 2009;Passonneau et al., 2012b).