However, in our situation, there is not always one single winner (two methods can perform similarly well for a given request), and there are different ways to pick winners (for example, based on F-score or precision).
such an approach would require the utilization of subjective heuristics for creating labels, which would significantly influence what is being learned.