When the search space is large, the policy may fail to sample a correct program, which can greatly slow down the learning.
off-policy methods are sometimes introduced to bias the sampling towards high-reward yielding programs.