The annotations in this data set were collected using Amazon Mechanical Turk, which often raises questions about the reliability of the data.
as described in Graham, Mathur, and Baldwin (2015), a rigorous quality control was conducted in order to filter out Average quality score and standard deviation of the scores assigned to the same sentences by different annotators.