It is worth mentioning that we have tried to initialize our Transformer model with pre-trained BERT, and then fine-tuning on WSJ-train.
in this setting, even when the training loss becomes lower than the loss of training from scratch, the parsing result is still far from our best results.