In order to encourage the generation of relevant captions, we propose a novel greedy algorithm that aims to minimize the cross entropy loss only for the most relevant and helpful gold-standard captions.
helpfulness is measured using the inner-product of the gradients from the caption generation loss and the VQA answer prediction loss.