More advanced gradient estimators exist that also reduce variance (Grathwohl et al., 2017;Tucker et al., 2017;Liu et al., 2018), but for simplicity we use the self-critical baseline method (Rennie et al., 2016), where the baseline is b(x) = l ω (y + , y , x), or b(x) = −s ω (y , x) in the separable loss case, and y = argmax i g θ (y i |x).
the baseline is the reward of the most likely sample according to the generator.