Yet, having a good estimate for the parameters in M w is essential for the final LM performance, as they are directly responsible for the next-word prediction.
our aim is to improve the quality of representations in M w , focusing on infrequent words.