Ideally, we should pre-train BERT 6 [Large] and BERT 6 [Base] from scratch, and use the weights learned from the pretraining step for weight initialization in KD training.
due to computational limits of training BERT 6 from scratch, we only initialize the student model with the first six layers of BERT 12 or BERT 24 .