When using ELMo, we first encode the sentence: , and we use the average of contextualized hidden states corresponding to the entity span as the contextualized entity representation.
f ELMo (w 1: With BERT, following Onoe and Durrett (2019), we concatenate the full sentence with the entity mention, starting with [CLS] and separating the two by [SEP], i.e., [CLS] We encode the full sequence using BERT and use the output from the [CLS] token as the entity mention representation.