That is, Equation 2 becomes: where [...;...] denotes concatenation.
at each layer l, the gate T l can regulate either passing through y l−1 to form y l , or retrieving novel knowledge from the input text x to augment y l−1 to create a better representation for y l .