Finally, evaluation with metrics is entirely reproducible.
despite the cost-effectiveness and other appealing aspects of automatic metrics in shared tasks, we do not believe that shared tasks in NLG should use automatic metrics as the sole evaluation criterion.