Many attempts have been paid to develop better neural language models (Zilly et al., 2017;Inan et al., 2016;Godin et al., 2017;Melis et al., 2017).
with extensive corpora, language models can be well trained to generate high-quality sentences from scratch (Józefowicz et al., 2016;Grave et al., 2017;Li et al., 2018;Shazeer et al., 2017).