We then explore several remedies and modifications to propose tractable and efficient NCE training strategies.
we propose to make the scaling factor a trainable parameter of the model, and to use the noise distribution to initialize the output bias.