Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks.
it is unclear why the pretraining-then-fine-tuning paradigm can improve performance and generalization capability across different tasks.