A high compression rate for large-scale language models is difficult to achieve on downstream NLP tasks.
there are few works in exploring and optimizing hardware-friendly model compression techniques for state-of-the-art Transformer-based pre-trained language models, to reduce the weight storage and computation on computer system while maintaining prediction accuracy.