The values for learning rate, learning rate shrinking and mini-batch sizes as well as context size are fixed once and for all based on insights drawn from previous work (Hai Son et al., 2011;Sukhbaatar et al., 2015;Devlin et al., 2014) as well as experimentation with the Penn Treebank validation set.
the learning rate is set to 0.05, with mini-batch size of 128 (we do not take the average of loss over the batch, and the training set is shuffled).