The reader may notice that if we apply delayed gradient updates, the effective batch size that the global optimizer deals with is τ times larger than the mini-batch size on which the local optimizers runs.
adam (Kingma and Ba, 2015) is used to perform asynchronous SGD with learning rate of 0.0001.