The human evaluator also conducted an extensive comparison of the baseline word-level models with the hierarchical CNN models.
she looked at those questions where the performance of both model types differed most (in terms of the number of models out of 11 that solved the question correctly).