The performance is impressive, with some tasks yielding lower perplexity than BERT pre-training + single task finetuning.
it still lags significantly behind fine-tuning applied after pushshift.io Reddit pretraining.