This situation indicates that considering multiple senses only in the pre-trained stage would add noise to the word representations.
the word representation tends to favor the commonly used senses in the corpora, which will disturb the model when the correct sense of the current word is not the common one.