The relative assessment is often done using the paired comparison (Thurstone, 1994) or the best-worst scaling (Woodworth and G, 1991;Louviere et al., 2015), to improve inter-annotator agreement.
absolute assessment of summarization (Li et al., 2018b;Song et al., 2018;Kryściński et al., 2018;Hsu et al., 2018;Hardy and Vlachos, 2018) is often done using the Likert rating scale (Likert, 1932) where a summary is assessed on a numerical scale.