BERT is also a transformer encoder that has access to the entire input but this choice requires a special training regime.
examples in BooksCorpus are a mix of individual sentences and paragraphs; examples are on average 36 tokens.