Neural models, with various neural architectures (Bengio et al., 2001;Mikolov et al., 2010;Chelba et al., 2014;Józefowicz et al., 2016), have recently achieved great success.
most of these neural architectures have a common issue: large output vocabularies cause a computational bottleneck due to the output normalization.