It seems that pre-training decoders has a smaller effect on the final performance than pre-training encoders (Lample and Conneau, 2019), one reason for which could be that the encoder-to-decoder attention is not pre-trained.
the parameters of the decoder need to be re-adjusted substantially in the following tuning process for MT task.