We also evaluate Turker translation quality by using them as reference sets to score various submissions to the NIST MT evaluation.
the second method selects the translation that received the best average rank, using the rank labels assigned by other turkers (see 3.3).