Our approach begins by first retrieving the set of sentences X such that each sentence contains at least one referring expression that refers to an entity of the type we are doing perturbation on (person, in our case).
whether the PSA assumption holds in individual sentences will depend on the sentential context; however, the corpus-level trends that we measure in the model scores/labels are still indicative of systemic biases in the model.