The most important advantage of pre-training is the possibility of using unlabeled data, which is really helpful given the high costs of data annotation.
the effect of pre-training with very large datasets observed in the experiments is the most surprising result of the paper (Erhan et al., 2010): the early examples determine the basin of attraction for the remainder of the training and the supervised fine-tuning cannot escape from it.