Recently pre-trained language models like OpenAI GPT (Radford et al., 2018), BERT (Devlin et al., 2018), ULM-FiT (Howard and Ruder, 2018) have made a breakthrough in a wide range of NLP tasks.
bERT and its variant models have shown superior performance in the GLUE benchmark for Natural Language Understanding (NLU) (Wang et al., 2019).