This approach only utilizes existing image-caption datasets, automatically determining which captions are relevant to a given question.
we design the training algorithm to only update the network parameters in the optimization process when the caption generation and VQA tasks agree on the direction of change.