In this paper, we propose a new knowledge distillation method, self-knowledge distillation (SKD) based on the word embedding of the training model itself.
self-knowledge is distilled from the predicted probabilities produced by the training model, expecting the model has more information as it is more trained.