Instead of waiting until the end of the label assignment process and comparing the predicted labels with the gold labels, we use reward shaping (Mao et al., 2018), i.e., giving intermediate rewards at each time step, to accelerate the learning process.
we set the reward r of x i at time step t to be the difference of Example-based F1 scores between current and the last time step: .