We randomly sampled 50 news abstracts from the test set and asked three native-speaker annotators for evaluation to score the generated headlines.
we conduct two tasks to evaluate on four criteria: (1) relevance, (2) attractiveness, (3) language fluency, and (4) style strength.