In other words, we face the following dilemma: • On the one hand, we should choose a small set ANN (i.e., select few annotators to base our collective annotation on), as that will allow us to increase the (average) reliability of the annotators taken into account.
• we should choose a large set ANN (i.e., select many annotators to base our collective annotation on), as that will increase the amount of information exploited.