Moreover, the counterintuitive sensitivity of SEAT on some models and biases suggests that biases revealed by SEAT may not generalize beyond the specific words and sentences in our test data.
our results invalidate the assumption that each set of words or sentences in our tests represents a coherent concept/attribute (like African American or pleasant) to the sentence encoders; hence, we do not assume the encoders will exhibit similar behavior on other potential elements of those concepts/attributes (other words or sentences representing, for example, African American or pleasant).