They further use a pre-trained language model as a prior for their compression model to induce their compressed output to be grammatical.
their reported results are still based on models trained on at least 500k instances of paired data.