HIBERT aims to learn the representation of a document, where its basic units are sentences.
a natural way of pre-training a document level model (e.g., HIBERT) is to predict a sentence (or sentences) instead of a word (or words).