Hence, we apply several adjustments to the hyperparameters to accommodate the larger effective batch size of multinode synchronous SGD: Learning rate: The Adam optimizer is scaleinvariant, so the parameters move at the same magnitude regardless the gradient size.
we linearly scale the learning rate in all multi-node experiments, as suggested by Goyal et al.