The models are finetuned using a small learning rate of 5e-5 and converge to a good solution fairly quickly as noticed by recent work (Li et al., 2020;Kaplan et al., 2020).
each experiment completed within a day of training on a single GPU, and many experiments with small datasets took a lot less time.