The error then increases as the model starts to overfit and knowledge captured through pretraining is lost.
uLMFiT is more stable and suffers from no such catastrophic forgetting; performance remains similar or improves until late epochs, which shows the positive effect of the learning rate schedule.