F SGD is the SGD update rule with the fixed learning rate γ max such that γ max := sup{γ i where γ i ∈ γ γ γ}.
for the SGD update rule F SGD , the fixed learning rate γ max is derived from the ADF update rule.