For time step i in layer j, the hidden state h i,j is calculated by employing self-attention over the hidden states in layer j − 1, which are {h 1,j−1 , h 2,j−1 , ..., h m,j−1 }, where m is the number of source-side words.
h i,j is calculated as follows: First, a self-attention sub-layer is employed to encode the context.