In this paper, we explored an alternate architecture for embedding-based neural network models which allows for a fully pre-computable network with multiple hidden layers.
two algorithms have been proposed to achieve this: (1) noise-contrastive estimation (NCE) (Mnih and teh, 2012;Vaswani et al., 2013) and (2) explicit self-normalization (Devlin et al., 2014), which is used in this paper.