The tokenizer simply splits on white-space and punctuation.
the POS tagger and lemmatizer components were developed in a separate project (Eiselen and Puttkammer,5 the FoLiA XML as used by the Frog parser; see http://ilk.uvt.nl/folia.