Based on our pilot study, we would expect annotators to agree to some degree, but we would not want such agreement to arise because annotators were collaborating.
we also exclude these annotations where we suspect the possibility of collaboration.