The error rate for 500 questions when a difference of 0.05 in confidence-weighted scores is observed is approximately 8 %.
if we know nothing about two systems except their scores which differ by 0.05, and if we repeat the experiment on 100 different sets of 500 questions, then on average we can expect 8 out of those 100 sets to favor one system while the remaining 92 to favor the other.