It is challenging to quantify the performance of RL-based neural generation systems because it is possible for a model to achieve high task reward and yet not generate human language (Das et al., 2017).
we propose a novel measure, the Language Constrained Reward (LCR) curve as an additional robust measure.