The tokenizer comprises a small set of manually developed finite-state rules for identifying word and sentence boundaries.
we also investigate use of the one-senseper-discourse heuristic to propagate a sense tag for a word to other occurrences of the same word within the current document in order to increase coverage.