To utilize the unlabelled data, we extend the traditional encoder-decoder model by leveraging two independent encoders but with some layers shared.
we share the weights of the last few layers of two encoders that are responsible for extracting high-level representations of input sentences.