We also removed pairs where one of the words was quite rare (fewer than 150 tokens in the entire corpus) or where, based on examples pulled from the corpus, there was a common confounding homonym-for instance the word prob, which is a common clipped form of both problem and probably.
the agreement is surprisingly low; even if "I'm not sure" responses are discounted, agreement with the writer's choice gold standard is just 71.7% for the remaining datapoints.