In contrast to other applications, like vision, where a single input can cover most or all of the model parameters, word embeddings are unique to each word, so a word will have its embedding updated only when the word appears in a training instance.
only words that appear in the labeled data will benefit from finetuning and, by changing only part of the embedding space, the performance may be worse overall.