Define the global frequency of k-Freq(k)-as the number of times k is chosen by someone, divided by the total number of individual annotations.
if Freq i (k) is high, particularly if Freq i (k) > Freq(k), we may want to give a relatively low weight to any instance of annotator i choosing catgeory k. Every BCR defines a family of weights w ik , specifying for each annotator i ∈ N and each category k ∈ K how much weight to give to i's choice of k: In case of a tie, we assign category 'undecided'.