On the other hand, fine-tuning approaches mainly pre-train a language model (e.g., GPT (Radford et al., 2018), BERT (Devlin et al., 2018)) on a large corpus with an unsupervised objective, and then fine-tune the model with in-domain labeled data for downstream applications (Dai and Le, 2015;Howard and Ruder, 2018).
bERT is a large-scale language model consisting of multiple layers of Transformer blocks (Vaswani et al., 2017).