We define this distillation for a sentence as, where V is the target vocabulary set.
this process is nonideal for two reasons: (1) unlike for standard knowledge distribution, it doubles the size of the training data, and (2) it requires training on both the teachergenerated sequence and the true sequence, conditioned on the same source input.