Standardly used LM models often have two layers of LSTM cells.
the issue we identifed might be mitigated in practice, since the hidden state of the first layer is not directly affected by tying the output and input transformation matrices.