This is a step towards having pretraining objectives that explicitly consider and leverage discourse-level relationships.
it is still unclear whether language modeling is the most effective method of pretrained language representation, especially for tasks that need to exploit multi-turn dependencies, e.g.