Different from previous knowledge distillation methods (Hinton et al., 2015;Sau and Balasubramanian, 2016;Lu et al., 2017), we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from previous layers of the teacher network.
once the network has been trained, there will be a high degree of parameter redundancy.