The first component, tokenization, can be treated with hand-written grammars, but this may be brittle and challenging to maintain.
it seems desirable to use machine-learned taggers, similar to those used to segment text in writing systems that do not separate words with spaces; see, for example, Wang and Xu (2017), who use convolutional neural networks for Chinese word segmentation.