As for a task such as ours one cannot really formulate test questions, which form CrowdFlower's main mechanism of quality control, we made use of a number of alternative control mechanisms that are commonly used with translationrelated tasks (Irvine and Klementiev, 2010; Zaidan and Callison-Burch, 2011): Generally, a good strategy for discouraging people from cheating is to design one's task in a way that makes cheating as laborious and time-consuming as actually working on the job at hand.
we showed the English seed sentences -and, in case of the later evaluation stage, the Japanese candidate translations -to the workers in the form of images rather than text, so as to make it more difficult for people to make use of automatic translation services.