While section 2 reviews two widely used training criteria for un-normalized model, Importance Sampling and NCE, section 3 provides an empirical analysis of the NCE.
the main noticeable result is that learning Z c as a parameter is not working well when we have less data -in this case, initializing the output bias to − log |V| is far more efficient, while initializing output bias with values of log P n (w) gives even better results.