be an RNN, s an input string of length n and 0 ≤ t ≤ n a time step.
we define • the hidden state vector h s,t ∈ Q N given by where h s,−1 = h −1 and we use standard matrix product and point-wise vector addition, Finally, the RNN R computes the weighted language R : Σ * → R, which is given for every in- each component h s,t (n) of the hidden state vector is the ReLU activation applied to a linear combination of all the components of the previous hidden state vector h s,t−1 together with a summand w st that depends on the t-th input letter s t .