We will show below that this pretraining is critical for getting good performance in the paraphrase task.
the general design principle of this type of unsupervised pretraining should be widely applicable given that next-word prediction training is possible in many NLP applications.