If the count is above the threshold, we preserve the full entry (for entries of length 3 or greater) only if the following inequality is true for each subsegment w j,k in the full entry w 1,n : That is, the ratio (expressed as a difference of logarithms) between the count of the segment and the full unsegmented entry (in our preliminary lexicon) is lower than the ratio of the predictability (as defined in our discussion of prediction-based decomposition) of the words in the segment with the context of the full entry to the predictability of words with only the context included within the segment (which is just pred(w j,k )).
we preserve only longer multiword sequences in our lexicon when any decrease in the probability of the full entry relative to its smaller components 2 is fully offset by an increase in the conditional probability of the individual words of that segment when the larger context from the full segment is available.