While Koh and Liang (2017) show that influence functions can be a good approximation even when the convexity assumption is not satisfied (in their case, a CNN for image classification), it is still not obvious that the influence function would work for BERT.
McCoy et al. (2019) hypothesize that the main artifact NLI models might learn is lexical overlap.