For our sampling-based objectives, we use k = 1024 samples, and the unigram distribution as p n .
2 may be simplified as following: We can further accelerate learning by using importance sampling on the second term, and because of the logarithm applied to the sum, we obtain an objective that is equivalent to composing the approximated softmax with γ-divergence, as can be seen in Appendix A.4.