We jointly train the embeddings of images and sentences together into the multi-modal vector space to build vision-specific word embeddings (Kiros et al., 2014b).
let T be the training dataset where one image I i is associated with a corresponding caption sentence S i , i.e., (I i , S i ) ∈ T .