We use the Jensen-Shannon Divergence (JSD) to quantify the difference between two attention distributions: We empirically characterize the relationship between attention weights and corresponding feature importance scores.
we measure correlations between attention and: (1) gradient based measures of feature importance (τ g ), and, differences in model output induced by leaving features out (τ loo ).