In text classification, Nguyen (2018) shows that automatic evaluation based on word deletion moderately correlate with human-grounded evaluations that ask crowdworkers to infer machine predictions based on explanations.
explanations that help humans infer machine predictions may not actually help humans make better decisions/predictions.