In order to measure the degree of agreement between annotators, we used one of the most commonly applied metrics in NLP research (Carletta, 1996): Kappa.
fleiss's kappa coefficient κ (fleiss, 1971), since there are three annotators for each annotation task as described above.