We have focused on whitespace tokenization and proposed a whitespacetokenization-free method that computes embeddings of higher quality than tokenization-based methods.
there are many properties of edited text beyond whitespace tokenization that a complex rule-based tokenizer exploits.