A challenge here is that we do not know the samples before performing the policy-gradient method.
we find that there is a high chance to get the same samples repeatedly when the model is pre-trained using MLE on the training set, because sampling is following the distribution learned by the pre-trained model.