Instead, the evaluation scheme, similar to the pyramid scheme used for summarization evaluation, assigns scores to each example based on its level of agreement among the annotators.
each example is assigned positive and negative scores, p = n + /N and n = n -/N, where n + is the number of annotators that annotate the example as positive, and nfor the negative.