In this paper, we argue that while we incorporate the prior knowledge with contextbased embeddings, words with different occurrences should be treated differently.
we propose to rely on the measurement of information content to control the degree of applying prior knowledge into context-based embeddings-different words would have different learning rates when adjusting their embeddings.