In machine translation, subword tokenization with byte pair encoding (BPE) addresses the problem of unknown words and improves performance (Sennrich et al., 2016).
segmentation is potentially ambiguous, and it is unclear whether preset tokenization offers the best performance for target tasks.