While REINFORCE assumes a stochastic policy, we use beam search for gradient estimation.
in contrast with common practice of approximating the gradient by sampling from the model, we use the top-k action sequences (programs) in the beam with normalized probabilities.