If stored using the same precision, the optimizer memory should be three times of model memory.
to validate our claim that BlockBERt with n Ã¢â€¡Â¥ n blocks can reduce the O(N 2 ) memory usage by a factor of n, we perform the same memory profiling as described in sections 2.1 and 2.2.