Few Words Annotated -When very few words in a document are annotated, we suspect the annotator may have been completing the task as quickly as possible and paying little attention.
we required that each annotator's work include enough annotated words to total more than 5% of the document's length.