Moreover, aside couple exceptions, it is always the same two raters that are excluded.
what we call 'best-pair kappa' is actually simply the removal from the data set of the annotation of the worse two raters (which is a relatively standard practice).