Due to the discrete choice of words in skeletons, the loss is no longer differentiable over the skeleton extraction module.
we use policy gradient (Sutton et al., 1999) to train the skeleton extraction module.