The lower MT scores and slower learning curve of the MTurk systems are both due to the lower quality of the translations, and to the mismatch with the professional development set translations (we discuss this issue further in §4.3).
we also show that adding a Mechanical Turk reference translation of the development set improves parameter tuning and output evaluation.