That is, we cannot extract any knowledge from the training model at the beginning.
we start training process without knowledge distillation at first and gradually increase the amount of knowledge distillation as the training iteration goes.