Additionally, the gold standard is itself imperfect, with typical estimates of professional disagreement around five percent.
we judge the quality of Mechanical Turk data by comparing the performance of one LVCSR system trained on Turker annotation and another trained on professional transcriptions of the same dataset.