We show that a multiple-choice version of Hot-potQA is vulnerable to the same baseline that performs well on WikiHop, showing that this distinction may be important from an evaluation standpoint.
looking at the no-context baseline for comparison, we find that it is only around 2% lower than the two relatively more complex models.