This intuition was supported by a short evaluation in which we sampled 314 (10.3%) random snippets from this set and found none that had been misclassified.
the negative examples of our Gold set consist of the snippets manually re-classified by our expert annotators, and the snippets which had been classified with 100% confidence by the crowd-workers.