As our experiments will show, under NCE with translation noise, the NNJM no longer provides meaningful scores for the entire vocabulary.
we also experiment with a novel mixture noise distribution: We deviate from their configuration by using a single 512-node hidden layer, motivated by our internal development experiments.