Training BERT is a memory-intensive process.
when n = 2, we denote 10:2 to be the configuration which assigns 10 heads to permutation (1, 2) and 2 to permutation (2, 1); when n = 3, we denote 8:2:2 to be the configuration which assigns 8, 2, 2 heads to permutation (1, 2, 3), (2, 3, 1), and (3, 1, 2), respectively.