A natural idea is to combine our graph model with pretrained LMs so as to combine the advantages of them.
the tokenlevel transformer encoding layer of our model can be replaced by a pre-trained LM like BERT.