To manage this issue we apply kernel smoothing over the language model to adapt it from the full vocab V down to the candidate word list C. Our smoothing process focuses on the output embeddings from the pretrained language model.
first we form the Voronoi partition (Aurenham-mer, 1991) over all the embeddings using the candidate set C. each word type w in the full vocabulary V is exactly assigned to one region represented by a word type w in the candidate set C, such that the distance from w to w is not greater than its distance to any other word types in C. As above, we use cosine similarity between corresponding word embeddings to define the regions.