Following Lee et al., we leverage a pretrained Stacked Cross Attention Neural Network (SCAN) to build a multi-modal semantic space.
we obtain word features by averaging the forward and backward hidden states per word from a bidirectional GRU (Bahdanau et al., 2014), where τ = {C, R} denotes either a candidate C or a reference R sentence of M words.