We construct a supervised training criterion by having our network predict sentences occurring next to each other in the training data.
for a pair of sentences (s i , s j ), we define a probability p(s i , s j ) that reflects how likely it is for the sentences to be adjacent to one another in the training data.