for j := 1 to minibatch size : for t := 0 to T −1 : Intervene: Evaluate each action at s t forā t ∈ A(s t ) : 11: 12: Improve: Train with dataset aggregation 13: Finalize: Pick the best policy over all iterations encountered when running the new policy π i+1 (or when parsing test sentences).
beyond just regularizing the training objective, we apply dataset aggregation : we take the training set to include not just Q i but also the examples from previous iterations (line 13).