However, 20 raters are often used in psychological studies (Warriner et al., 2013;Stadthagen-Gonzalez et al., 2017b), while being way higher than the number of raters typically used in NLP for emotion annotation, both for the word and sentence level (Yu et al., 2016a;Strapparava and Mihalcea, 2007).
we argue that this choice constitutes a rather challenging line of comparison for our system.