One way to handle this problem is to build a language model of content tokens and retain only the maximum likelihood token sequence.
in the current work, the following heuristic which worked well in practice is used.