Language models (LMs) based on sequential LSTMs (Hochreiter and Schmidhuber, 1997) have numerous practical applications, but it has also been shown that they do not always develop accurate syntactic generalisations (Marvin and Linzen, 2018).
one strategy for improving LSTMs is to change their biases to facilitate more linguistically valid generalisations.