When selecting the activation function, since we expect the final predicted value of the scaling factors to cover the range of 0∼1, we applied the Sigmoid activation function to the original outputs of the scaling gate.
the scaling gate takes the x ∈ R h and the F(x, W) ∈ R h as input and computes the output through two linear transformations with a Tanh activation in between: are the parameters to be learned, and S(x, F) is followed by a Sigmoid activation function.