We found that with our approach we can learn additional entity knowledge in BERT that helps in entity linking.
we also found that almost none of the downstream tasks really required entity knowledge, which is an interesting observation and an open question for future research.