The main idea is (i) to automatically select a subset of less-noisy examples applying an automatic classifier, (ii) training the annotators with such subset, and (iii) iterating this process after retraining the classifiers using the annotated data.
the educated crowd workers can provide higher quality annotations, which can be used by the system in the next iteration to improve the quality of its classification.