Furthermore, since some data instances that are collected using distant supervision are noisy, we annotated the correctness of the test instances using Amazon Mechanical Turk annotations to perform a fair comparison.
we asked workers to determine whether or not a given sentence expressed a given relation.