In the task settings for CrowdFlower, we specified that we needed annotations from six people for each word.
11 because of the way the gold questions work in CrowdFlower, they were annotated by more than six people.