To mitigate this issue, instead of forcing the student to learn only from the logits of the last layer, we propose a "patient" teacher-student mechanism to distill knowledge from the teacher's intermediate layers as well.
we investigate two patient distillation strategies: (i) PKD-Skip: the student learns from every k layers of the teacher (Figure 1: Left); and (ii) PKD-Last: the student learns from the last k layers of the teacher (Figure 1: Right).