The knowledge learned about the document during the pre-training process will be transferred and benefit on the summarization task.
the Mask task randomly masks some sentences and predicts the missing sentence from a candidate pool; the Replace task randomly replaces some sentences with sentences from other documents and predicts if a sentence is replaced.