It is worth noting that pretraining on SQuAD-T (Table 2) yields 0.5 point lower MAP than pretraining on 50% of SQuAD.
roughly speaking, span-level supervision data is worth more than twice the size of sentence-level supervision data for the purpose of pretraining.