Recently Lin and Parikh (2016) have leveraged visual question answering models to encode images and descriptions into the same space.
all of this work is targeted at monolingual descriptions, i.e., mapping images and descriptions in a single language onto a joint embedding space.