In addition, we leveraged crowdsourcing to check whether we could use (untrained) people to evaluate VA candidates.
workers on the crowdsourcing platform Figure Eight had to check whether a sentence contained a VA. 10 Crowdsourcing resulted in 600 judgments for 200 randomly selected candidates of our baseline data set with an inter-annotator agreement calculated by Cohen's Kappa of 0.72 between expert and workers.