In contrast, we propose an input representation which allows the Transformer layers in BERT to directly leverage contextualized representations of all words in all sentences, while still utilizing the pretrained weights from BERT.
we represent all the sentences in the document as one long sequence of words with special delimiter tokens in between them.