As RNNs model sequential data by a fully-connected layer, dropout can be applied by simply dropping the previous hidden state of a network.
we modify Equation 1 in the following way: where d is the dropout function defined as follows: where p is the dropout rate and mask is a vector, sampled from the Bernoulli distribution with success probability 1 − p. Long Short-Term Memory networks (Hochreiter and Schmidhuber, 1997) have introduced the concept of gated inputs in RNNs, which effectively allow the network to preserve its memory over a larger number of time steps during both forward and backward passes, thus alleviating the problem of vanishing gradients (Bengio et al., 1994).