These pre-trained models, particularly the ones based on the Transformer architecture (Vaswani et al., 2017) 1 have achieved state-of-the-art results in a variety of NLP tasks, but come at a great cost financially and environmentally (Strubell et al., 2019;Schwartz et al., 2019).
cross-View Training (cVT) ) is a semi-supervised approach that uses unlabeled data in a task-specific manner, rather than trying to learn general representations that can be used for many downstream tasks.