It also impacts the quantity of training data required, and the training stability.
considering these properties, we investigate the impact of the noise distribution on the training of language models.