Let us denote this expected cumulative probability distribution for flat segmentation as P d (x; |q|) = the probability that for a pair of randomly chosen flat segmentations of q, q m and q n , d(q m , q n ) ≥ x.
likewise, let P d 1 (x; |q|) and P d 2 (x; |q|) be the respective probabilities that for any two nested segmentations q m and q n of q, the following holds: We define the IAA under random bias model as (k is 1, 2 or null): S is the expected probability of observing a similar or worse agreement by random chance, averaged over all pairs of annotations for all queries, and not a chance corrected IAA metric such as α.