The comparison was automatic, measuring similarity among vectors of content words (i.e., stop words such as articles and prepositions were removed).
here is an example to further clarify the strict standard of the automatic evaluation (words scored correct are underlined): System: E. Lawrence Barcella is a Washington lawyer, Washington white-collar defense lawyer, former federal prosecutor System Merge: Washington white-collar defense lawyer human Merge: a Washington lawyer and former federal prosecutor Automatic Score: Correct=2; Extra-Words=2; Missed-Words=3 although 'lawyer' and 'prosecutor' are synonymous in WordNet, the automatic scorer doesn't know that, and so 'prosecutor' is penalized as an extra word.