State-of-the-art performance is usually obtained by fine-tuning the pre-trained model on the specific task.
bERT-based models are currently dominating the leaderboards for SQuAD 1 (Rajpurkar et al., 2016) and GLUE benchmarks 2 (Wang et al., 2018).