Thus, the distance between the teacher's prediction and the student's prediction can be defined as: where c is a class label and C denotes the set of class labels.
besides encouraging the student model to imitate the teacher's behavior, we can also fine-tune the student model on target tasks, where taskspecific cross-entropy loss is included for model training: the final objective function for knowledge distillation can be formulated as: where α is the hyper-parameter that balances the importance of the cross-entropy loss and the distillation loss.