We use an attention function Att, scale the scalar attention value using sof tmax, multiply it with the value representation M v i and sum the result into a single vector value representation c kn s i : (10) Att is a dot product, but it can be replaced with another attention function.
of this operation, the context token representation c ctx s i and the corresponding retrieved knowledge c kn s i are in the same vector space ∈ R 2h .