ADAGRAD is a popular learning rate method designed for online optimization problems with high dimension and sparse gradients.
the learning rates will not monotonically decrease, but vary in a certain range.