In order to address this issue, they computed the translation-bytranslation correlation with human assessments (i.e., correlation at the sentence level).
correlation with human judgements is not enough to determine the reliability of measures.