With the aid of the likelihood ratio trick, the gradient for the dialog policy is In the fashion of AL, the reward estimator aims to distinguish real human sessions and generated sessions from the dialog policy.
it minimizes KL-divergence with the empirical distribution, while maximizing the KL-divergence with the policy distribution, Similarly, H(p) and H(π θ ) is independent to ω, so the gradient for the reward estimator yields So far, the reward estimation uses the entire session τ , which can be very inefficient because of reward sparsity and may be of high variance due to the different lengths of sessions.