This problem is however a difficult problem on its own and often requires decades of research (see, e.g., Creutz and Lagus (2005) for Finnish and other morphologically rich languages and Huang and Zhao (2007) for Chinese).
many opt to using either a rule-based tokenization approach or a suboptimal, but still available, learning based segmentation algorithm.