As shown in Figure 1, when designing the model, the hidden size in each layer is much smaller than the hidden size of the corresponding layer in the residual-connected model.
while each encoder layer perceives information from its previous layers, each decoder layer z l+1 has two information sources: previous layers z i , i ≤ l, and attention values a i , i ≤ l. in order to allow dense information flow, we redefine the generation of (l+1)-th layer as a nonlinear function over all its previous decoder layers and previous attentions.