The pre-trained language model is then adapted to downstream tasks and further fine-tuned.
the top layer of the language model can be replaced by a task-specific layer and then continue to train on downstream tasks.