Recurrent Neural Networks (RNNs) have shown impressive performances on many sequential modeling tasks due to their ability to encode unbounded input histories.
training simple RNNs is difficult because of the vanishing and exploding gradient problems (Bengio et al., 1994;Pascanu et al., 2013).