On the CrowdFlower task settings, we specified that we needed annotations from ten people for each instance.
because of the way the gold instances are setup, they are annotated by more than ten people.