As mentioned, noisy, largescale datasets with user-generated captions exist for news images (Berg et al., 2004;Feng and Lapata, 2008) and Flickr (Ordonez et al., 2011;Chen et al., 2015;Thomee et al., 2015).
in this paper, we are mainly interested in literal descriptions of what is depicted in the image, rather than non-literal or non-visual descriptions that require significant inference from additional knowledge about the image context.