The PTB corpus uses the standard vocabulary of 10k words, and for the ASPEC corpus we use a vocabulary of the 20k most frequent words.
the model learns to set the mixture proportions of the δ elements to near zero and rely mainly on the count-based n-gram distributions.