The attention score a q j,i on memory m q i ∈ M q for a RNN hidden state t j in the decoder part is computed as follows: where W q 1 and W q 2 are weight vectors associated with m q i and t j , respectively.
the attention vector on the query sequences is given by: Similarly, the attention scores and attention vectors on the other three memories can be derived by replacing q with r, c, and h in Equations 14, 15.