Apparently, strict matches on surface forms seems reasonable for extractive summarization datasets.
we still see that our word mover metrics, i.e., WMD-1+BERT+MNLI+PMeans, perform better than or come close to even the supervised metric S 3 best .