While some pre-trained models trained with web texts or Wikipedia are public, our preliminary results indicate that fine-tuning of a pre-trained model trained with the same corpus has a better performance.
we performed pre-training from scratch.