The goal of this paper is to analyze whether we can distill sense-specific information from the pretrained contextualized word embeddings such as ELMo and BERT for better interpretation.
given the embedding of a (word, context) pair, our model learns a non-linear mapping network f : X → Y to project it into the desired  Pre-Trained BERT Figure 1: Illustration of the proposed model using contextualized embeddings from BERT as the context-dependent component.