Third, we normalize the key vectors using the softmax function: where ∈ ℝ 1 , ∈ ℝ and the bias is added to each element of ∈ ℝ 1 .
the weights in dictate to what extent the network should attend each document.