The attention scores a i are determined by a dot product between h i with each z j , followed by a softmax over the source sequence: In preliminary experiments, we did not find the MLP attention of (Bahdanau et al., 2015) to perform significantly better in terms of BLEU nor perplexity.
we found the dot-product attention to be more favorable in terms of training and evaluation speed.