Following the standard REINFORCE algorithm (Williams, 1992), the gradient for the expectation E n,sn∼π θ [R(s n )] in Equation 16 can be written as: is joint negative loglikelihood of sentence s n and its indicator idx n , and it can be decomposed as: where L AW , L N W and L P L are negative loglikelihoods; p AW , p N W and p P L are probabilities of taking an action; 1 denotes indicator function.
equation 17 can be re-written as: It is very hard to train agents using reinforcement learning from scratch, therefore a good initialization for policy network is usually required (Bahdanau et al., 2016;Silver et al., 2016;Wang et al., 2018b).