In addition to automatic evaluation using ROUGE, we also evaluated system output by eliciting human judgments.
we assessed the degree to which our model retains key information from the document following a questionanswering (QA) paradigm which has been previously used to evaluate summary quality and document compression (Clarke and Lapata, 2010;Narayan et al., 2018b).