Considering that the reward for different words in a sentence y t should be different, we further propose to use the reward at the word level as follows: It can be found that the classifier-based discriminator only provides reward for the finished sequence.
for a sequence of length T , to evaluate the action-value for a word at the time step t, Monte Carlo Search (MCS) with a roll-out policy G θ is usually applied to sample the unknown last T − t tokens (Yu et al., 2017).