That is, while we can easily maximise agreement, doing so in a naïve way means ignoring most of the information collected.
we face the following dilemma: • On the one hand, we should choose a small set ANN (i.e., select few annotators to base our collective annotation on), as that will allow us to increase the (average) reliability of the annotators taken into account.