First, however, we make the following critical point: It is possible for a model trained on (l (i) , q IID from p(l, q, d) to exceed the performance of a single annotator on this measure.
if we have a model p (l|q, d; θ), trained on (l, q, d) triples, which is a good approximation to p (l|q, d), it is then possible to use p (l|q, d; θ) to make predictions that outperform a single random draw from p (l|q, d).