Formally, has the form: where t is the iteration number, T is the total number of training iterations, f is a monotonically increasing function, and we introduce two new hyper-parameters associated with the cyclical annealing schedule: • M : number of cycles (default M = 4); • R: proportion used to increase within a cycle (default R = 0.5).
this issue causes two undesirable outcomes: (i) an encoder that produces posteriors almost identical to the Gaussian prior, for all observations (rather than a more interesting posterior); and (ii) a decoder that completely ignores the latent variable z, and a learned model that reduces to a simpler language model.