While some work (Luong et al., 2015b;Luong and Manning, 2016;Wu et al., 2016) has achieved state-of-the-art results using deep networks, others (Jean et al., 2015;Chung et al., 2016;Sennrich et al., 2016b) have produced similar results with far shallower ones.
we hope that this allows us to isolate the effect of various hyperparameter changes.