If we let W ∈ R n×n and U ∈ R n×m then the total number of parameters in the model (excluding the bias terms) is 4(nm+n 2 ) which grows quadratically as n grows.
increases in LSTM size can substantially increase the number of parameters.