To be able to fit the computational graph of the document NMT model within GPU memory limits, we pre-train the sentence-level bidirectional RNN using the language modelling training objective.
the document-level bidirectional RNN is trained together with other parameters of the document NMT model by back-propagating the document translation training objective.