To get this desired behavior, we employ regularization to maximize the likelihood of each token in the training data independently (as if reset every step).
we call this a unigram regularizer: and we add it to the training objective: the overall training loss is: L = L text + L U .