More importantly, a tokenization close to a gold standard does not necessarily leads to better translation quality (Chang et al., 2008;Zhang et al., 2008).
it is necessary to offer more tokenizations to SMT systems to alleviate the tokenization error propagation problem.