In the previous sections, we have assumed that the input sequence is pre-segmented into tokens.
in real-world applications the normalization model needs to be fed with segmented output produced by a segmenter.