On the one hand, posing an over strong en- tropy regularization can easily cause the actor to diverge.
the model diverges when τ reaches 0.03 on the image captioning task or 0.06 on the machine translation task.