In all three models, the higher the layer, the lower the self-similarity is on average.
given that upper layers are much more contextspecific than lower layers, and given that gPT-2's representations are more context-specific than ELMo and BERT's (see Figure 2), this suggests that the PCs of highly context-specific representations are less effective on traditional benchmarks.