Moreover, an LSTM cell carries over information both through the hidden state and a memory state; the latter is affected by tying only indirectly (see Hochreiter and Schmidhuber (1997) for details on LSTM architectures).
our experiments show that, in practice, two-layer LSTM LMs are still affected by tying despite these caveats.