This figure shows that a softmax model with 1024 hidden units in the last layer could perform better than the 512-hidden unit model with a longer training horizon.
for neural models, we train 11-gram models for gigaword and bil-lionW; for PTB we train a 6-gram model.