Note that the agent needs to take tens of actions (with the number depending on the length ofŷ) to generate a complete sentenceŷ, but only one reward is available for all those actions.
rL training is inefficient due to the sparsity of rewards, and the model updates each token in the training sentence with the same reward value without distinction.