Concurrent work on model CHECKLIST evaluation (Ribeiro et al., 2020) includes an invariance test which also examines model undersensitivity.
to CHECKLIST, our work focuses with more detail on the analysis of the invariance phenomenon, the automatic generation of probing samples, an investigation of concrete methods to overcome undesirably invariant model behaviour, and shows that adherence to invariance tests leads to more robust model generalisation.