Different from Zhao et al., 2018) which used only the last hidden state h q tq as the feature embedding for the query sentence, we generate visually guided sentence features H qp = {h qp t } tp t=1 by exploiting their fine-grained relationships based on H q and H p .
given the i-th visual feature h p i , an attention mechanism (Xu et al., 2015) is used to adaptively summarize where .