The reward r is non-zero only at the last step of decoding, and is the F 1 score computed comparing the gold answer and the answer generated by executing the program a 0:T .
the cumulative reward of a program a 0:T is The agent's decision making procedure at each time is defined by a policy, ⇡ ✓ (s, a) = P ✓ (a t = a|x, a 0:t 1 ), where ✓ are the model parameters.