If the rare words are not initialized properly, they would also bias the whole word embeddings.
we more incline to make character-level embedding to represent a rare word, and build CWE embeddings for others.