Our pruning experiments also reveal some patterns in the distribution of redundancy in NMT.
we find that higher layers, attention and softmax weights are the most important, while lower layers and the embedding weights hold a lot of redundancy.