One may think that it is sufficient to simply enumerate frequent-enough words or N -grams and use them as a whitelist to avoid suppressing their occurrences.
lCP i is defined as the length of the maximum common prefix of two lexicographically neighboring suffixes T SA i 1 ..n and T SA i ..n , where T is the string in concern and SA is the suffix array of T .