Note that, for both cache pointer and dynamic evaluation, we coarsely tune the associated hyperparameters on the validation set.
with negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method improves perplexity on the Penn Treebank dataset by up to 1.8 points and by up to 2.3 points on the wikiText-2 dataset, over strong regularized baselines using a single softmax.