Furthermore, we perform experiments with a conventional NMT framework, in which a part of the initial value of a weight matrix is set to zero so that the matrix is at the same initial-state as the key-value attention mechanism.
we obtain comparable results with the key-value attention mechanism without changing the network structure.