The best-performing PC static embeddings belong to the first layer of BERT, although those from the other layers of BERT and ELMo also outperform GloVe and FastText on most benchmarks.
in all layers of all three models, the contextualized word representations of all words are not isotropic: they are not uniformly distributed with respect to direction.