This raises an issue with the evaluation; a metric can look strong when evaluated on current model outputs, but may in-fact struggle in the future when QA systems produce better answers.
using only these candidates for evaluation could lead to overconfidence in a learned metric's capabilities.