Normally, the pre-training procedures are designed to learn on tokens corresponding to small units of texts (e.g., word pieces for English, characters for Chinese) for efficiency and simplicity.
some important information carried by larger text units may be lost for certain languages when we use a standard encoder, such as BERT.