Recent work (Tenney et al., 2019b;Liu et al., 2019a;Goldberg, 2019;Tenney et al., 2019a) has shown that the pre-trained models can encode syntactic and semantic information of language.
it is unclear why pre-training * Contribution during internship at Microsoft Research.