As the gold standard of performing human evaluation is often too expensive, there has been a large effort develop-ing automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin and Rey, 2004), ME-TEOR (Lavie and Denkowski, 2009;Denkowski and Lavie, 2014) and CiDER (Vedantam et al., 2015).
next, we look at evaluating the correctness of system outputs in question answering using the MS MARCO question answering dataset (nguyen et al., 2016).