Experimental results show that the gradient-based approach outperforms both the best black-box method and other comparative methods.
• We provide empirical support for the design principle of NMT architectures: essential inductive bias (e.g., language characteristics) should be considered for model design.