x t ∈ R l is an input vector of length l. The most common approach is to use an RNN such that: where h t ∈ R n is a hidden state at time t, and f is generally a non-linear transformation, producing e := h T +1 as the input representation.
the decoder is trained to predict the next output y t given the encoded input vector e and all the previously predicted outputs the decoder defines a probability over the output sequence t by decomposing the joint probability into ordered conditionals: With a decoder RNN, we can first obtain the hidden layer at time t as: s t = g(s t−1 , {e, y t−1 }) and feed this into a softmax layer to obtain the conditional probability as: where, In recent work, both f and g are generally LStMs, a kind of RNN which we describe next.