One fundamental design criterion for our annotation system was the ability to select examples in real time to support, e.g., the interactive annotation experiments presented in this paper.
we could not use the standard MTurk workflow or services like CrowdFlower.