(2018) and remove workers who demonstrate consistently low correlations with others' judgments.
for each sentence pair s, for each worker w i , we compute the Spearman correlation between w i 's labels and 4 Raters do not see specific numbers on the slider.