Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks.
for all three contextualizing models, PC static embeddings created from lower layers are more effective those created from upper layers.