However, the cross-lingual information is obtained mostly from the shared BPE space during their pre-training method, which is inexplicit and limited.
we figure out a new pre-training method that gives the model more explicit and stronger cross-lingual information.