A standard Elman network (Elman, 1990) with hidden layer of size H trained on a corpus of size T with vocabulary of size V has complexity where I indicates the number of iterations.
several attempts have been made to reduce training time, focusing mostly on reducing the large factors T or V : • vocabulary shortlisting (schwenk and Gauvain, 2004) • subsampling (schwenk and Gauvain, 2005;Xu et al., 2011) • class-based (Goodman, 2001b;Morin and Bengio, 2005;Mikolov et al., 2011) • noise-contrastive estimation (Gutmann and Hyvärinen, 2012;Chen et al., 2015) these techniques either come with a serious performance degradation (Le et al., 2013) or do not sufficiently speed up training.