GQA (Hudson and Manning, 2019) uses real scenes from Visual Genome, but the language is artificially generated.
we found this to be surprisingly competitive on VCR compared to published baselines, perhaps due to our choice of powerful pretrained models.