10 This isn't quite accurate as the annotators are sampled without replacement; however, it simplifies the analysis.
11 for an input (q, d), if we define , d), then the Bayes optimal hypothesis is to output l * if P (h β (a, l * ) = 1|γ,γ) ≥ P (h β (a, NULL) = 1|γ,γ), and to output NULL otherwise.