In our query expansion step, we swap modalities and construct a new query based on the distributed representations of captions.
we expand the original visual query with a new textual query based on the weighted average of the vectors of the retrieved captions as follows: where N and M respectively denote the total number of image-caption pairs in the candidate set N and the number of reference captions associated with each training image, and sim(I q , I i ) refers to the visual similarity score of the image I i to the 1 The adaptive neighborhood parameter was emprically set to 0.15. query image I q 2 which is used to give more importance to the captions of images visually more close to the query image.