After the seq2seq model M is initialized with the two LMs, it is fine-tuned with a labeled dataset.
this procedure may lead to catastrophic forgetting, where the model's performance on the language modeling tasks falls dramatically after fine-tuning (Goodfellow et al., 2013).