This suggests that BERT is aware of the structure of the document it is given.
we observe a decent accuracy in identifying discourse relations between adjacent EDUs, perhaps due to the "next sentence prediction" task in pre-training, as pointed out in (Shi and Demberg, 2019).