We detect those by inspecting annotated texts for sources of disagreement between annotators, and redefining the annotation scheme so that these disagreements are minimized.
in order to assess the reproducibility of human annotations, we used Cohen's kappa (Cohen, 1960).