We also estimated the chance agreement p c in the usual way as: where c 1 and c 2 are the proportions of tokens annotated with the given class in the first and second annotations respectively, i.e., c 1 = A 1 T and c 2 = A 2 T .
we are currently testing our annotation guidelines on a new group of undergraduate students to see whether the provided descriptions generalize to other coders as well.