For pre-training, we use the same data as BERT, which consists of 3.3 Billion tokens from Wikipedia and BooksCorpus (Zhu et al., 2015).
this also indicates that splitting the pruning for attention heads could have more flexibility for the model to find an optimal architecture.