More interestingly, we find that performance can be improved if the system scores and human ratings are aggregated over several topic cardinalities before computing the correlation.
2014proposed an automated approach to the word intrusion task.