Note that the manual judgments that indicate a document as relevant or non-relevant relative to a given query contain disagreements (Carterette et al., 2008;Voorhees, 2000) and errors (Alonso and Mizzaro, 2012).
a 64% agreement (cf.