We also computed the number of floating point operations (FLOPs) in the model's forward path as well as the number of total parameters for all architectures (cf.
we first increase the learning rate linearly during the number of warmup steps, keep it a constant until the decay start step s, then exponentially decay until the decay end step e, and keep it at 5 • 10 −5 after the decay ends.