Since the appearance and rise of attentionbased models, many work naturally inspect attention scores and interpret with them.
we are aware of the recent discussion over whether attention is a kind of faithful explanation (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).