The encoder reads through X and computes a sequence of hidden states h 1:Tx : where F enc is a non-linear activation function (e.g., LSTM), h t is the hidden state at time t. Each time the model generates a target word y t , the decoder looks at a set of positions in the source sentence where the most relevant information is located.
another non-linear activation function F dec is used for the decoder where the hidden state s t at time t is computed by: Here, the context vector c t is computed as a weighted sum of the hidden vectors h 1:Tx : where is the dot product of two vectors.