Past work has reported great variation in how human evaluations are done (van der Lee et al., 2019).
we begin with a meta-analysis of a subset of human evaluation experiments from EMNLP 2019, which we then use as the basis for claims about the power of human evaluations in NLP more generally.