In theory, it allows for stronger "self-normalization" during training (Arora et al., 2017), and in practice, subtracting the mean vector from static embeddings leads to improvements on several downstream NLP tasks (Mu et al., 2018).
the extreme degree of anisotropy seen in contextualized word representations -particularly in higher layersis surprising.