Because our goal is to investigate whether a model can learn to search through a document, it is important that a non-negligible fraction of the questions require navigation through the document.
in Wikipedia each document starts with a preface that summarizes the document, and thus often contains the answer.