The above findings prove that BERT has learned its own syntax as a by-product of self-supervised training, not by directly copying any human design.
giving the superior performance of BERT on downstream tasks, it is natural to ask if BERT is learning an empirically useful structure of language.