We are currently investigating whether we can achieve better reliability with fewer responses per item by taking into account an individual annotator's history across items, as recent work suggests is possible (Passonneau and Carpenter, 2013;Rzhetsky et al., 2009;Whitehill et al., 2009).
crowd-sourcing VerbNet semantic entailments appears to be both feasible and productive.