The probability is learnt during training according to the following equation: ) Then the next word y t is generated according to: The joint probability is decomposed as: p g is the probability of generating a word from the vocabulary and is defined according to Equation 7. p c is the probability of copying a word from the source vector x and is defined as the sum of the word's attention weights.
the probability of copying a word x is defined as: Decoder coverage In long sequences, the neural generation models tend to repeat phrases where the softmax layer predicts the same phrase multiple times over multiple timesteps.