Question answering tasks are widely used for training and testing machine comprehension and reasoning (Rajpurkar et al., 2016;Joshi et al., 2017).
high performance in standard automatic metrics has been achieved with only superficial understanding, as models exploit simple correlations in the data that happen to be predictive on most test examples.