Neural network models have achieved remarkable performance on text classification due to their capacity of representation learning on natural language texts (Zhang et al., 2015;Yang et al., 2016;Joulin et al., 2017;Devlin et al., 2018).
the lack of understanding of their prediction behaviors has become a critical issue for reliability and trustworthiness and hindered their applications in the real world (Lipton, 2016;Ribeiro et al., 2016;Jacovi and Goldberg, 2020).