Their success can be attributed to neural gates that adaptively read or discard information to/from internal memory states.
a LSTM network successively reads the input token x t , internal state c t−1 , as well as the visible state h t−1 , and generates the new states c t , h t : where i, f and o are input, forget and output gates, respectively.