Moreover, even using a language-specific tokenizer might not always solve this problem, because some tokenization ambiguities cannot be solved other than by an arbitrary choice depending on interpretation; for instance, the documentation about the way tokenization is carried out for French in the UD2 corpus mentions that: "This tokenizing and segmentating choice is arbitrary and other French treebanks could choose to do otherwise."
many NLP shared tasks nowadays provide datasets annotated with the linguistic information relevant to the task, so that participants can focus on the core aspects of the task rather than spend time on non-essential pre-processing steps.