The comparison in Table 1 indicates that the best performing setting comes from 12 selfattention layers, and more than 12 layers shows almost no promotion even reduces the accuracy.
the rest experiments are done with 12 layers of the self-attention encoder.