Initially, manual evaluation was carried out, where human judges were tasked to assess the quality of automatically generated summaries.
in an effort to make evaluation more scaleable, the automatic ROUGE 1 measure (Lin, 2004b) was introduced in DUC-2004.