The maximum-likelihood training objective is the minimization of the loss: We use a policy gradient method (Williams, 1992;Sutton et al., 2000) to calculate the second term in the objective function.
we sample a generated response for each input sequence (conversation history) x, and assign to it a reward R, which in our case is the politeness classifier's probability of the response classified as polite.