This prevents models from doing better or worse just because they received different starting points.
three preference judgments are obtained for each pair of translations and are combined using weighted majority vote.