The Document Understanding Conference (DUC 2002) run by the National Institute of Standards and Technology (NIST) sets out to address this problem by providing annual large scale common evaluations in text summarization.
these evaluations involve human judges and hence are subject to variability (Rath et al.