A recent analysis of approaches to evaluating compression (Napoles et al., 2011b) has shown a strong correlation between the compression rate and human judgments of compression quality, thereby concluding that comparisons of systems which compress at different rates are unreliable.
all comparisons that we carry out here involve a restriction to a particular compression rate to ensure that observed differences can be interpreted meaningfully.