Moreover, the commonly used evaluation metric ROUGE yields a noisy surrogate evaluation (despite its good correlation with human judgments) compared to the much more meaningful evaluation based on human judgments.
the current end-toend evaluation does not provide any insights into the task of automatic summarization.