Even fine-tuning the pre-trained model with taskspecific dataset may take several hours to finish one epoch.
ideally, we should pre-train BERT 6 [Large] and BERT 6 [Base] from scratch, and use the weights learned from the pretraining step for weight initialization in KD training.