Furthermore, the results on row 4 (HMLSTM + Schedule + LN) are better than the reproduction results which also implement the CopyLast feature.
in the rest of the experiments, we consistently add layer normalization on all layers as well as learning-rate scheduling, and drop CopyLast.