There are two reasons: i) Information compressed in image features can be partially retrieved from neighboring pixels since spatially they share similar and uniform characteristics, whereas syntax and semantics information in deep Transformer in language/text domain are not uniformly characterized; ii) Intuitively, the high-level semantic, syntax, and language understanding capability might be broken when we prune zero or near-zero weights in the latent space.
a high compression rate for large-scale language models is difficult to achieve on downstream NLP tasks.