A crucial difference distinguishing our work from the STILTs, Snorkel MeTaL, and MT-DNN KD methods in Table 2 is that we do not single-task fine-tune our model.
we do not continue training the model on individual tasks after multi-task training.