We propose automatic procedures to revise VQA and Visual7W (and to create new datasets based on COCOQA (Ren et al., 2015), VQA2 (Goyal et al., 2017), and Visual Genome) such that the decoy generation is carefully orchestrated to prevent learning algorithms from exploiting the shortcuts in the datasets by overfitting on incidental statistics.
our design goal is that a learning machine needs to understand all the 3 components of an image-question-candidate answers triplet in order to make the right choiceignoring either one or two components will result in drastic degradation in performance.