Given that isotropy has benefits for static embeddings, it may also have benefits for contextualized word representations, although the latter have already yielded significant improvements despite being highly anisotropic.
adding an anisotropy penalty to the language modelling objective -to encourage the contextualized representations to be more isotropic -may yield even better results.