For a word w ∈ C, we shall learn a sparse vector x = (x 1 , x 2 , .
the number of parameters in the Embedding and Prediction subnets in neural LMs increases linearly with respect to the vocabulary size, which is large (table 1).