GRUs have a linear shortcut through timesteps which bypasses the nonlinearity and thus promotes gradient flow.
a GRU computes the hidden state at current time step, h t , as the linear combination of previous activation h t−1 , and a new candidate activationh t : where is elementwise multiplication, and the update gate activation z t determines the amount of new information mixed in the current state: The candidate activation is computed as: The reset gate r t determines how much of the current input x t is mixed in the previous state h t−1 to form the candidate activation: By applying the gru function repeatedly a GRU layer maps a sequence of inputs to a sequence of states: where X stands for the matrix composed of input column vectors x 1 , .