The basic idea is to use an ROC-style curve to visualize the tradeoff between achieving higher reward and being faithful to human language.
at each checkpoint i over the course of RL training, we record two measures: (1) the PPL of a given model on the test data p i = PPL(θ i ) and (2) this model's average cumulative task reward in the test environment R t i .