The design of residual connection can provide us with a promising way to find identity mappings for the Transformer model.
residual connection (He et al., 2016b) can be formalized as H(x) = F(x) + x, where F could be either MHA or FFN and H is the sub-layer output.