Due to the large size of model parameters and deep architectures, modern neural networks training requires significant amounts of computing resources.
there is an increasing interest in training neural networks with low memory (Sohoni et al., 2019).