Given that our extractor performs a nondifferentiable hard extraction, we apply standard policy gradient methods to bridge the backpropagation and form an end-to-end trainable (stochastic) computation graph.
simply starting from a randomly initialized network to train the whole model in an end-to-end fashion is infeasible.