Note that on this task regularizing only the recurrent connections is more beneficial than only the forward ones.
lSTM networks trained with our approach and the approach of (Gal, 2015) yield a lower bit-per-character (bpc) score than those trained with forward dropout onlyWe attribute it to pronounced long term dependencies.