Recent advances in self-supervised pretraining (Radford et al., 2018; Devlin et al., 2018a; Liu et al., 2019) have resulted in impressive downstream performance on several NLP tasks (Wang et al., 2018, 2019).
this has led to the development of enormous models, which often require days of training on non-commodity hardware (e.g. TPUs)