Note that we framed the problem as a multi-label task, and considered two annotations to be identical if and only if their sets of assigned labels are identical.
the inter-annotator agreement estimate is probably somewhat pessimistic.