The parameters in our model are optimized with Adam (Kingma and Ba, 2015), which keeps a moving average of the L2 norm of the gradient for each parameter throughout training and divides the gradient for each parameter by this moving average, ensuring that the magnitude of the gradients will on average be close to one.
the k-order pruning does not boost the performance of our model.