Formally, instead of Equation 2, DOC computes the output probability distribution at timestep t + 1 by the following equation: where π j,ct is a weight for each probability distribution, k j,ct ∈ R d is a vector computed from each hidden state h n , andW ∈ R V ×d is a weight matrix.
p t+1 is the weighted average of J probability distributions.