Pairwise comparisons can be used to infer rankings or ratings by assuming a random utility model (Thurstone, 1927), meaning that the annotator chooses an instance with probability p, where p is a function of the utility of the instance.
when instances in a pair have similar utilities, the annotator selects one with a probability close to 0.5, while for instances with very different utilities, the instance with higher utility will be chosen consistently.