Advances in techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015;Luong et al., 2015;Dozat and Manning, 2017;Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018;Devlin et al., 2019;Radford et al., 2019;So et al., 2019).
training a state-of-the-art model now requires substantial computational resources which demand considerable energy, along with the associated financial and environmental costs.