One might be surprised the improved results by KVQ-FH since the model sizes of KVQ-FH were relatively very small comparing with the original embeddings.
here, we consider utilizing a squared loss function Ψ lsq , which can be written as the summation of the squared losses over an individual embedding vector e w : where C w is a weight factor for each word.