Note that recognizing multiword tokens is quite different from tokenization -the multiword token is usually not just a simple string concatenation.
the described tokenizer does not handle multiword tokens.