This weight is the negative log probability of the resulting model after adding the corresponding features and priors.
the lexicon prior poses a problem for this construction since the penalty incurred by a new unit in the segmentation depends on whether that unit is present elsewhere in that segmentation.