The Information Bottleneck (IB) method has been successfully applied to clustering in the NLP domain (Slonim and Tishby, 2000).
iB attempts to balance the trade-off between accuracy and compression (or complexity) while clustering the target variable, given a joint probability distribution between the target variable and an ob-served relevant variable.