We add these tags to the vocabulary and random initialize their embeddings in the embedding layer, thus the input can be described as following: where E t , E p , E li represent the token embedding matrix, position embedding matrix and linguistic feature embedding matrix, C l is the collection of all types of linguistic features and w wt , w wp , w wli are the corresponding weight scalars.
because of the differences between language model and NLU task objectives, the problem of lack of concern on some key words exists.