Large models are trained with a dropout rate of 0.4 until the validation performance stopped improving (250K steps for text8 and 150K steps for enwik8), and then further trained for 20K steps with a learning rate divided by 10.
in small models, we apply dropout with a rate of 0.3 to the attention and the feedforward ReLU activations.