In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting.
we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process.