We attribute it to post-norm Transformer being more sensitive to the large learning rate.
in the case of either a 6-layer encoder or a 20-layer encoder, the pre-norm Transformer benefits from the larger batch and learning rate.