It is important to note that the decision for N * = 20 is necessarily arbitrary, to some degree, with higher SHR estimates arising from higher values of N * .
20 raters are often used in psychological studies (Warriner et al., 2013;Stadthagen-Gonzalez et al., 2017b), while being way higher than the number of raters typically used in NLP for emotion annotation, both for the word and sentence level (Yu et al., 2016a;Strapparava and Mihalcea, 2007).