When pretraining using a LM objective, models are typically trained until model fit stops improving (i.e., perplexity on validation data).
other work has suggest that random initializations have a significant effect on the failure cases of BERT, pointing to the brittleness of fine-tuning (Phang et al., 2018).