When working with a recurrent unit that has no internal bias term, this behavior is entirely determined by the bias term of the final sigmoid output layer, σ(wx + b), which with typical random initialization of b results in a default predicted value of roughly 0.5.
this 0.5 default value is not always optimal or semantically appropriate to the predictive task.