These methods include bi-directional language modeling (Peters et al., 2018), masked language models (Devlin et al., 2019), word order permutation (Yang et al., 2019), more robust training (Liu et al., 2019) and more efficient architectures (Lan et al., 2019).
little focus has been put on learning discourse coherence as part of the pretraining objective.