We also propose two different strategies for the distillation process: (i) PKD-Last: the student learns from the last k layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and (ii) PKD-Skip: the student learns from every k layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.
the distance between the teacher's prediction and the student's prediction can be defined as: where c is a class label and C denotes the set of class labels.