This observation is consistent with many previous works of using RL/IL in NLP.
we also notice that the improvement of using the REINFORCE algorithm (line 6) is very small, only 0.18 on the average score.