To address these challenges, we propose a simple and effective paraphrase-aware retrofitting (PAR) method that is applicable to arbitrary pretrained contextualized embeddings.
pAR prepends an orthogonal transformation layer to a contextualized embedding model.