It should be noted that exponential decay is not a good choice from a theoretical point of view, because it does not satisfy one of the necessary con-ditions for convergence-the sum of the learning rates must diverge to infinity (Spall, 2005).
this is probably not a big issue for practitioners because normally the training has to be terminated at a certain number of iterations in practice.