Having built one s + for each layer from the given text x, the activation of the transform gate for layer l (l > 0) (i.e., Equation 3) is calculated: To generate feature representation y l , the function f takes as input the concatenation of y l−1 and s + l .
equation 2 becomes: where [...;...] denotes concatenation.