With the rapid development of research on Neural Machine Translation (NMT), translation quality has been improved significantly compared with traditional statistical based method (Bahdanau et al., 2014;Cho et al., 2015;Zhou et al., 2016;Sennrich et al., 2015).
training efficiency is one of the main challenges for both academia and industry.