There exist several studies for compressing neural language models.
quantization (Lin et al., 2016;Hubara et al., 2016) and groupwise low rank approximation (Chen et al., 2018a) achieve the state-of-the-art performance on tasks like language modeling and machine translation.