Question answering from text (Richardson et al., 2013;Hill et al., 2015;Hermann et al., 2015;Rajpurkar et al., 2016) is a key challenge problem for NLP that tests whether models can extract information based on a query.
even sophisticated models that perform well on QA benchmarks (Seo et al., 2017;Shen et al., 2017; may only be doing shallow pattern matching of the question against the supporting passage (Weissenborn et al., 2017).