At a high level, the regularization term keeps parameters which are important to general-domain performance close to the initial general-domain model values during continued training, while allowing parameters less important to general-domain performance to adapt more aggressively to the in-domain data.
the first, multi-objective fine-tuning, which they denote MCL, trains the network with a joint objective of standard loglikelihood loss plus a second term based on knowledge distillation (Hinton et al., 2015;Kim and Rush, 2016) of the general-domain model.