(2017a) on CrowdFlower in order to evaluate whether lay annotators suffice for a theory-based quality assessment.
we asked the crowd to rate the same 304 arguments as the experts for all 15 given quality dimensions with scores from 1 to 3 (or choose "cannot judge").