This architecture gives strong results on wordlevel language modeling while reducing the number of parameters significantly.
pRU improves the perplexity of a recent state-of-the-art language model Merity et al.