We chose hyper-parameters based on a grid search and picked the one which gave the best perplexity on the validation set.
we searched over the number of hidden units H of the recurrent layer, the learning rate η, the learning rate annealing schedule γ (the factor by which to decrease η if the validation perplexity increases), and the gradient clipping threshold κ.