This extra layer seems to be crucial for improving performance on this task.
we refer to encoding hidden layers as e, and decoding hidden layers as d. The PN uses a form of content-based attention (Bahdanau et al., 2014) to allow the model to produce a distribution over input elements.