Next, given the visual grounding of constituents, Θ is optimized for producing constituents that can be better matched with images.
we optimize textual representations and the visual-semantic embedding space using a hinge-based triplet ranking loss: where i and k index over all image-caption pairs in the data set, while j and enumerate all constituents of a specific caption (c (i) and j } is the set of textual representations of all constituents, and δ is a constant margin, The loss L extends the loss for image-caption retrieval of Kiros et al.