Unlike in COCO, the lack of a particular object label in an image for Visual Genome does not necessarily indicate that the object is not present, both due to annotation noise and the use of multiple synonyms for objects by human labelers.
we restrict the set of candidate irrelevant images to those which contain a matching object to the question premise but a different attribute.