Moreover, even using a language-specific tokenizer might not always solve this problem, because some tokenization ambiguities cannot be solved other than by an arbitrary choice depending on interpretation; for instance, the documentation about the way tokenization is carried out for French in the UD2 corpus mentions that: "This tokenizing and segmentating choice is arbitrary and other French treebanks could choose to do otherwise."
18 a dataset-specific tokenizer should be trained on the provided input data in order to tokenize the third-party corpus accurately.