One of us then used the Kuwait consensus agreement to annotate the 16 machine summaries for that text which were created by different participants in DUC-2002, an annotation which could be done rather quickly.
voorhees (2000) shows that the relative rankings of IR systems are stable across annotators even though relevance judgements differ significantly between humans.