Since we use the reference summaries as target sequences to train the model and assume that they are the gold standard, we give both articles and reference summaries to the annotator to score the generated summaries.
relevance  we compare the generated summaries against the reference ones and the original article to obtain the (relative) scores in Table 3.