Thus it is natural to use trained languages models to initialize the encoder and decoder.
the basic procedure of our approach is to pretrain both the seq2seq encoder and decoder networks with language models, which can be trained on large amounts of unlabeled text data.