It also tells us which Word-level model PTB WikiText-2 Small Medium Small Medium LSTM + ND (Jozefowicz et al., 2015) -79.8 --LSTM + ND (Kim et al., 2016) 97.6 85.4 116.8 † -LSTM + ND (Zaremba et al., 2014  Removing dropout of the context state: According to (1), context state s t changes linearly and thus should not suffer from over-fitting.
it seems reasonable to try to not regularize it and apply dropout only to the hidden states, i.e.