Our proposed Past Decode Regularization method builds on the work of using sophisticated regularization techniques to train LSTMs for language modeling.
the AWD-LSTM model achieves state-of-the-art performance with a single softmax on the four datasets considered in this paper (Merity et al., 2018b,a).