They note that all the metrics assume that a fixed set of labels are used for items.
we adapt the equations presented in pp.565-566 of Artstein and Poesio (2008) for measuring these quantities for queries: is a distance metric for the agreement between annotations q m and q ′ n .