Recently, Recurrent Neural Networks LMs (RNN-LMs) have became the state-of-the-art approach to LMs (Józefowicz et al., 2016).
rNN-LMs struggle to keep their level of performance as the length of the input increases.