In particular, the WebAnno we are using, makes it possible to calculate Kohen's Kappa, Fleiss' Kappa and Krippendorff's Alpha.
in order to do it properly, a tool needs to know all interdependencies among categories of tags.