To deal with this, we use a training method we call staged back-propagation which saves memory by considering slices of K tokens in z, rather than the full sequence.
to train on a minibatch where |z| = 300 with K = 100, we would actually train on 3 mini-batches, where the first batch would optimize for the first z 1:100 , the second for z 101:200 and the third for z 201:300 .