This bias also appears in pool-based evaluation for knowledge base population (Chaganty et al., 2017).
the problems with automatic metrics clearly motivate the need for human evaluation, but can we still use the automatic metrics somehow to save costs?