Instead of using the point estimate ∂R c ∂φ of the gradient of the predicted objective with respect to the actor's parameters φ, we propose to use the expected gradient of the predicted objective with respect to the critic-aware distribution Q.
where we define the critic-aware distribution Q as .