We use two-layer attention for illustration, while the approach is applicable to any layers lower than l. transforming information across layers.
for constructing each hidden state in any layer-l, we allow the self-attention model to attend any layers lower than l, instead of just layer l-1: where C l −i is sequential vectors queried from layer l-i using a separate attention model, and AGG(•) is similar to the pre-defined aggregation function to transform k vectors {C l −1 , .