It turned out, however, that a 3-Binomial model with a good fit on one of the samples was rejected with high probability for the other.
1 on the one hand, the variations between annotators in each sample were not as high as to preclude a model that captures only instance variability from fitting well; on the other hand, evidently, the two samples did not come from the same annotator distribution, but differed systematically due to factors we did not control for.