In this group of experiments, we investigated the impacts of out-of-domain corpus size on our proposed framework.
we conjecture that during the process of knowledge distillation, by assigning non-zero probabilities to multiple words, the output distribution of teacher model is more smooth, leading to smaller variance in gradients (Hinton et al., 2015).