3 We ask non-expert workers to provide annotations for POV, a difficult decision for a subset of sentences.
the design of the HIT (Human Intelligence Task) in AMT is crucial: in order to get acceptable agreement, the AMT task must be simple and easy to understand; definitions must be clear and the annotation interface well-structured.