There exists an RNN with saturated linear activation, input alphabet Σ = {a}, and 1 designated neuron n ∈ N such that for all s ∈ Σ * and 0 ≤ t ≤ |s| • h s,t (n) = 0 if M does not halt on ε, and • if M does halt on empty input after T steps, then h s,t (n) = 1 if t = T 0 otherwise.
such RNNs with saturated linear activation can semi-decide halting of an arbitrary Turing machine in the sense that a particular neuron achieves value 1 at some point during the evolution if and only if the Turing machine halts on empty input.