Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences.
we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation.