The sense pairs (250 positive and 250 negative pairs) were uniformly sampled from sense clusters obtained using the same automatic method described above, excluding the sense pairs that were included in the automatically created datasets.
there is no overlap between the 500 sense pairs in the manually annotated datasets, and the 6,106 (5,000) sense pairs in the automatically created datasets.