In addition to calculating scores on a human vs human basis, we also calculated the F-scores for the top three CoNLL-2014 teams, AMU (Junczys-Dowmunt and Grundkiewicz, 2014), CAMB (Felice et al., 2014), and CUUI (Rozovskaya et al., 2014), versus all the combinations of humans (Equation 3).
s ∈ S, where S is the set of all three shared task systems, i.e., {AMU, CAMB, CUUI}, and f (s, X) is the same function in Equation 1 which is the score computed by the M2 scorer to evaluate system s against the set of annotators X chosen to be in the gold standard.