Statistical methods like n-grams have also been extensively used to segment character sequences into words and have proven very effective (Kawtrakul, 1997;Aroonmanakun, 2002).
in the absence of word boundary markers, tokenization of text for further processing is a nontrivial and challenging task.