Besides, considering the chain rule in the backward propagation, the complexity of computing gradients for a hierarchical recurrent model increases exponentially from the top layer of the model down to the bottom layer.
to improve the training efficiency of hierarchical recurrent models, our strategy is to simplify the baseline models by making the lower layers simpler than the upper layers, which we name as "the lower the simpler".