A fully connected softmax layer is used as classifier for classification.
rNNs with the above form may suffer from gradient exploding or vanishing problem (Bengio et al., 1994;Hochreiter, 1997) during training when it is trained with the backpropagation through time algorithm (rumelhart et al., 1986;Werbos, 1990;Williams and Zipser, 1995).