To take into account all layers, we compute the weighted sum of all hidden layers, h l i , where 1 ≤ l ≤ L, corresponding to a word position i, through attention mechanism.
h i in Equation 7 is replaced by the following equation: where m ∈ R d model is a projection vector to obtain scalar values with the key vectors.