Even if the γ γ γ is initialized with high values of learning rates, after a number of training passes (denoted as T) γ γ γ T is guaranteed to be small enough so that γ max : Without losing generality, our convergence analysis starts from the pass T and we take γ γ γ T as γ γ γ 0 in the following analysis.
we can show that the ADF update rule with a decaying learning rate vector is a dynamic contraction mapping: Theorem 2 (ADF update rule with decaying learning rates) Let γ γ γ t be a learning rate vector in the ADF learning algorithm, which is decaying over the time t and with different decaying rates based on feature frequency information.