While the task in earlier QA tracks had already been validated, changes to the 2002 task were significant enough to warrant further examination.
the 2002 task required systems to return exact answers, to return one response per question, and to rank questions by trials overlap, there may be correlations among the trials that could bias the estimates of the error rates as compared to what would be obtained with an equal number of samples drawn from a much larger initial set of questions.