Average performance across datasets in Table 4, when choosing the best source dataset from Table 2, is 38.9 (DOCQA) and 43.5 (BERTQA).
one can select a dataset to pre-train on based on generalization performance and suffer a minimal hit in accuracy, without fine-tuning on each dataset.