timization methods such as stochastic gradient descent (SGD) (Robbins and Monro, 1951); stochastic methods of this type are particularly important for training with large data sets.
recently, recurrent neural networks (rNNs) have shown promising performance on this task (Mikolov et al., 2010;Sutskever et al., 2011).