The values M 3,i are converted to weights W of dimension [b, s, k, 1] using the formula W = max(0, 1 − M ) 3 , with the max operation ensuring the model puts a weight of zero on tokens in separate entities (see the reduction of the value of 1.7 in M in Figure 6 to a weighting of 0.0).
the weights are normalized to sum to 1, and multiplied with the unfolded token embeddings X to give the entity embeddings t , of dimension [b, s, e] the embeddings at the end of level 1 for the words "the", "United" and "Kingdom" (t 1 1 , t 1 2 and t 1 3 respectively) are all now close to equal, and all have been formed from a weighted average of the three separate token embeddings.