Naively, it might seem like we would then check how often A beats B by more than δ(x) on x (i) .
there's something seriously wrong with these x (i) as far as the null hypothesis is concerned: the x (i) were sampled from x, and so their average δ(x (i) ) won't be zero like the null hypothesis demands; the average will instead be around δ(x).