Following Theorem 2, we have the following formula for the ADF training: where γ max is the maximum learning rate at the beginning.
let d 0 := |w w w 2 − w w w 1 | and q := 1 − γ max /σ 2 , then we have: When Hence, we have: M t is upper-bounded.