To minimize the training cost, we test method 4 on a pre-trained BERT large model.
we load a pre-trained BERT large model, bert-large-uncased-whole-word-masking, from pytorch transformer 5 as the initial model and fine-tune the existing parameters and the new relative position embedding parameters for 3 epochs, staring with a small learning rate of 5e âˆ’ 5.