Our approach exhibits small GPU memory footprints, due to reductions of the arithmetic complexity (with fewer intermediate results to keep) and trainable parameter size (with fewer parameters to store).
training with continuous outputs is 2 to 4 times more memory-efficient than with the softmax layer (see Section 5.2).