One root cause of these problems is SQuAD's focus on questions for which a correct answer is guaranteed to exist in the context document.
models only need to select the span that seems most related to the question, instead of checking that the answer is actually entailed by the text.