Finally, we conduct experiments on formal languages, justifying that our theorems correctly predict which languages unsaturated recognizers trained by gradient descent can learn.
we view our hierarchy as a useful formal tool for understanding the relative capabilities of different RNN architectures.