We train the model end-to-end with only the traditional predictive language modeling perplexity objective.
our system can model named entities in text more accurately as demonstrated by reduced perplexities compared to traditional LM baselines.