Previous works (Fried et al., 2018;Dhingra et al., 2018) proposed to first pre-train the model with synthetic data and then fine-tune it with ground-truth data.
we find when the synthetic data size is small (e.g., similar size as the ground-truth data), catastrophic forgetting  will happen during fine-tuning, leading to similar results as using ground-truth data only.