We propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data.
after a dataset is constructed, we recommend that the dataset authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets.