We denote a segmentation of the N tokens of corpus w N 1 into M N segments by specifying segment boundary markers Setting a boundary marker s i =a means that we put a segment boundary before the a-th word token (or the end-of-corpus marker for a=N +1).
our corpus is segmented into token sequences j=1 .