The notable drop between development and test accuracy for SQuAD is a reflection of an artifact in the dataset-its 100k questions are derived from only 536 documents.
good retrieval targets are highly correlated between training examples, violating the IID assumption, and making it unsuitable for learned retrieval.