Some generic tokenization methods can be used as a fallback, but this is rarely optimal.
such cases can only be solved by training a tokenizer on the provided input data, in order to apply the same tokenization choices in the thirdparty corpus.