Pre-training these large language models has largely improved the state of the art of the GLUE benchmark (Wang et al., 2018).
our work uses the BERT pre-trained model and makes use of the representations obtained not only in the encoder but also on the decoder in a language generation task.