First, viewing passages of the same question as independent training instances may result in incomparable answer scores across passages.
globally normalizing scores over all passages of the same question (Clark and Gardner, 2018) may be helpful.