The large vocabulary sizes encountered in training corpora arguably stem from the fact that the frequency distribution of words in a corpus of natural language follows Zipf's law (Powers, 1998).
tracking these values during training 13 shows that they all behave very similarly to perplexity.