(2017) pretrain encoder-decoder pairs using language models and sequence autoencoders and then fine tune with task specific supervision.
after pretraining the biLM with unlabeled data, we fix the weights and add additional taskspecific model capacity, allowing us to leverage large, rich and universal biLM representations for cases where downstream training data size dictates a smaller supervised model.