The pretraining prepares BertS2S to be more aware of the domain of the document and less prone to language model vulnerabilities.
bertS2S is more confident in predicting tokens from the document than TranS2S, hence, improving faithfulness.