In our primitive experiments, we observed that training with larger batches and learning rates worked well for deep models.
all the results of deep models are reported with batch = 8192, lr = 2×10 −3 and warmup = 16,000 unless otherwise stated.