When annotating atomic phenomena, our experts identified and annotated the type, the scope, and in some paraphrase types, the key element.
when the annotation process involves more than two annotators, we first calculate the pairwise TPO measure between any two annotators and then we use one of three different techniques for calculating the overall agreement for the corpus.