To study our estimators offline, we preemptively collected 10,000 human judgments which cover several  1: (a) At a system-level, automatic metrics (ROUGE-L) and human judgment correlate well, but (b) the instance-level correlation plot (where each point is a system prediction) shows that the instancelevel correlation is quite low (ρ = 0.31).
if we try to locally improve systems to produce better answers ( in (a)), they do not significantly improve ROUGE scores and vice versa ( ).