Let p(w|c) be the probability of a word w given a context c that represents its entire preceding context, and let p(w) be a 'noise' word distribution (e.g.
we show that with minor modifications to word2vec's algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models.