The normalization constant is Z(u) = w∈V p(w | u), and V is the vocabulary.
the standard approach for estimating neural language models is maximum liklelihood estimation (MLE), where we learn the parameters θ * that maximize the likelihood of the training data, for each training instance, gradient-based approaches for MLE require a summation over all units in the output layer, one for each word in V .