(b) The model is first trained with step-by-step supervision, and then Gumbel-Softmax is applied to refine the policy.
sT-Gumbel is difficult to train (similar to RL), and may be stuck in poor local optima, resulting in low self-agreement for multiple random initializations (Williams et al., 2018a).