First we group the tokens into phrases, assigning to each token a single phrase greedily.
while lexicons have proven useful in various NLP tasks, a small amount of noise in a lexicon can severely impair the its usefulness as a feature in log-linear models.