We found that summarization system ranking, based on scores for multiple topics, was surprisingly stable and didn't change signifi-  cantly when several topics were removed from consideration.
on a summary level, removing topics scored by the most inconsistent assessors helped ROUGE-2 increase its correlation with human metrics.