We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy.
we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels.