For a worker, we compute his or her unreliability score as follows: for x in X(w) where w is a worker, U (w) is the unreliability score of w, X(w) is the set of annotations annotated by w (possibly over multiple news editorials), and Average(u(x)) for x in X(w) indicates the average of the unreliability scores of all the annotations conducted by the worker w. In this study, the annotators were asked to proceed with the annotation in one direction.
they were asked to read a sentence, to annotate the sentence, then to read the next sentence, and so on.