We also found that 67.5% of mistakes made by the CNN+LSTM model can be successfully predicted by checking distance of test QI pair from its k-NN training QI pairs (66.7% for the ATT model, 55.08% for the MCB model).
this analysis not only exposes a reason for mistakes made by VQA models, but also allows us to build human-like models that can predict their own oncoming failures, and potentially refuse to answer questions that are 'too different' from ones seen in past.