Once the source sequence is encoded, another decoding RNN model is to generate a target sequence through the following prediction model: where s t is the RNN hidden state at time t, the predicted target word y t at time t is typically performed by a sof tmax classifier over a settled vocabulary (e.g.
30,000 words) through function g. The prediction model of classical decoders for each target word y i share the same context vector c. a fixed vector is not enough to obtain a better result on generating a long targets.The attention mechanism in the decoding can dynamically choose context c t at each time step (Bahdanau et al., 2014), for example, representing c t as the weighted sum of the source where the function ρ use to compute the attentive strength with each source state, which usually adopts a neural network such as multi-layer perceptron (MLP).