It takes as input the BERT representations {h L i } output from the previous layer, and enriches them with relevant KB embeddings, which makes the representations not only context-aware but also knowledge-aware.
for each token s i , we get its BERT representation h L i ∈ R d 1 and retrieve a set of potentially relevant KB concepts C(s i ), where each concept c j is associated with KB embedding c j ∈ R d 2 .