The BERT model architecture consists of multiple layers of Transformers and uses a specific input representation, with two special tokens, [CLS] and [SEP], added at the beginning of the input sentence pair and between the sentences (or bag of sentences) respectively.
the semantics of a sentence are usually more dependent on local context, rather than all sentences in a long docu- ment.