Training with RL usually has 2 steps: supervised pretraining and policy gradient reinforcement learning (Williams and Zweig, 2016;Dhingra et al., 2017;Li et al., 2016).
the supervised learning step maximizes the log likelihood on the training dialogs, where θ is the model parameter: Then the following RL step uses policy gradients, e.g.