Such a joint training approach requires the model to continuously update the vocabulary predictor during RE-INFORCE, because the encoder is shared.
the action space for each input changes during reinforcement learning, and we observed unstable training.