Early approaches to deep semi-supervised learning pretrain neural models on unlabeled data, which has been successful for applications in computer vision (Jarrett et al., 2009;LeCun et al., 2010) and NLP.
noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011;Mikolov et al., 2013;Pennington et al., 2014) and language model pretraining (Dai and Le, 2015;Ramachandran et al., 2017;Peters et al., 2018;Howard and Ruder, 2018;Radford et al., 2018).