We also employ the same regularization strategy (DropConnect (Wan et al., 2013) + Dropout (Srivastava et al., 2014)) and weight tying approach.
we perform optimization using Adam (Kingma and Ba, 2015) with learning rate 1e-3 instead of NT-ASGD, having found that it is more stable.