Our algorithm ( § 4) follows recent work that has demonstrated the power of pre-training for improving DNN models in NLP (Peters et al., 2018;Devlin et al., 2019) and particularly for domain adaptation (Ziser and Reichart, 2018).
whereas in previous work a representation model, also known as a contextualized embedding model, is trained on a language modeling related task, our algorithm utilizes a representation model that is trained on sequence prediction tasks derived from the parser's output.