In segmented languages such as English, since words are seemingly separated by white spaces or punctuation marks, tokenization is regarded as a relatively easy task and little attention has been paid to.
each language dependent morphological analyzer has its own strategy for tokenization.