A subset of the disagreed annotation was manually adjudicated by experts, from which we found that taking the union of the entity sets annotated by two workers would effectively give the correct set of entities for each of those disagreed plural mentions.
a vast amount of plural mentions were pseudo-adjudicated by taking their unions of double-annotation.