Inspired by ULMFiT (Howard and Ruder, 2018), we propose a two-stage pre-training method to train our lattice language model.
applying these models to the spoken scenarios poses several discrepancies between the pre-training task and the target task, such as the domain mismatch between written texts and spoken utterances with ASR errors.