• We demonstrate the effectiveness of the approach on top of a strong system (Transformer) which has already used residual networks.
the attention model used in transformer is scaled dot-product attention 1 .