Likewise, let P d 1 (x; |q|) and P d 2 (x; |q|) be the respective probabilities that for any two nested segmentations q m and q n of q, the following holds: We define the IAA under random bias model as (k is 1, 2 or null): Thus, S is the expected probability of observing a similar or worse agreement by random chance, averaged over all pairs of annotations for all queries, and not a chance corrected IAA metric such as α.
s = 1 implies that the observed agreement is almost always better than that by random chance and s = 0.5 and 0 respectively imply that the observed agreement is as good as and almost always worse than that by random chance.