<unk>, effectively changing the vocabulary thus making perplexity incomparable to models without these factors, without improving WER noticeably.
we do plan to use more overt factors in Machine Translation experiments where a language model faces a wider range of OOV phenomena, such as abbreviations, foreign words, numbers, dates, time, etc.