We found that ROUGE scores can be fairly uncorrelated with human evaluation, and in general can be distorted by quirks of the data set or model outputs, particularly pertaining to length, formatting, and handling of special tokens.
human evaluation can be more sensitive to comprehensibility and relevancy while being more robust to rewording and reasonable ambiguity.