For evaluation of metrics that operate at the system or document-level such as BLEU, inconsistency in individual human judgments can, to some degree, be overcome by aggregation of individual human assessments over the segments within a document.
for evaluation of segment-level metrics, there is no escaping the need to boost the consistency of human annotation of individual segments.