These are used to build annotation pairs, s i , s j , which are positive instances if s i has a lower concept annotation error than s j , with respect to the manual annotation.
a trained binary classifier can decide if s i is more accurate than s j .