Empirically, LSTMs encode the most recent noun as the subject of a verb by default, but they are still capable of learning to encode grammatical inflection from the first word in a sequence rather than the most recent (Ravfogel et al., 2019).
while inductive biases inherent to the model play a critical role in the ability of an LSTM to learn effectively, they are neither necessary nor sufficient in determining what the model can learn.