Formally, has the form: where t is the iteration number, T is the total number of training iterations, f is a monotonically increasing function, and we introduce two new hyper-parameters associated with the cyclical annealing schedule: • M : number of cycles (default M = 4); • R: proportion used to increase within a cycle (default R = 0.5).
we split the training process into M cycles, each starting with = 0 and ending with = 1.