Different from previous work, ZEN provides an alternative way of learning larger granular text for pre-trained models, where the structure of BERT is extended by another Transformer-style encoder to represent the extracted n-grams for each input text instance.
The second step of n-gram extraction is performed during pre-training, where some n-grams in L are selected according to each training instance c = (c1, c2, ..., ci, ..., ckc) with kc characters.