One of the most reliable ways of comparing metrics is to measure their correlation with human judgements.
for the word-level QE task, asking humans to rate a system labelling or to compare the outputs of two or more QE systems is a very expensive process.