Previous approaches to obtaining such judgments include training annotators to assign scores from 1-6 (Persing and Ng, 2017), asking annotators for simple binary or three-class categories (Wei et al., 2016b), and aggregating binary votes from multiple people (Wei et al., 2016a;Tan et al., 2016).
these approaches are limited by the cost of training annotators, a highly restricted set of categories, or the need for multiple annotators per document.