Bohm et al. (2019) train a BERT-based evaluation function with 2,500 human ratings for 500 machinegenerated summaries from the CNN/DailyMail dataset; their method correlates better with human ratings than ROUGE and BLEU.
as their method is designed for evaluating single-document summaries, it correlates poorly with the Pyramid scores for multi-document summaries (see Â§3).