More importantly, the dataset size we want to annotate is quite large, hence it is too expensive to recruit in-house annotators for that task.
we used CrowdFlower crowdsourcing platform 9 to acquire   annotations.