We found that LSTM and GRU fail for all learning rates, EURNN is unstable for large learning rates, and RUM is stable for all learning rates.
we use 0.001 for all units except for EURNN, for which we use 0.0001.