We thus curate a subset of data instances for human annotation of comment quality, which is also used for enhancing automatic metrics as in the next section.
we randomly select a set of 1,610 articles such that each article has at least 30 com-ments, each of which contains more than 5 words, and has over 200 upvotes for its comments in total.