However, these methods are usually designed for written text, so they do not consider the properties of spoken language.
this paper aims at generalizing the idea of language model pre-training to lattices generated by recognition systems.