The behavior is justified by our optimized update procedure, in which the learning rate η k is used as a tradeoff parameter.
it tries a very aggressive update at the early stage of training, but eventually becomes conservative in updating parameters.