In the first case, the rank of the first correct item is the minimum rank of any item judged correct by either judge.
the phrase translation probabilities p(e 1 |f ) and p(f |e 2 ) are estimated by maximum likelihood estimation and smoothed using Good-turing smoothing (Foster et al., 2006).