The probability of generating next word e i is, 2 where t i is a decoding state for time step i, computed by, s i is a source representation for time i, calculated as, where α i,j scores how well the inputs around position j and the output at position i match, computed as, As we can see, NMT only learns an attention (alignment) distribution for each target word over all source words and does not provides exact mutually-exclusive word or phrase level alignments.
it is known that attentional NMT systems make mistakes in over-or undertranslation (Cohn et al., 2016;Mi et al., 2016).