We use the projected image as the first and/or last word of the source sentence and let the attention model learn when to attend to the image representation.
given the global image feature vector q ∈ R 4096 , we compute (6): where W 1 I ∈ R 4096×4096 and W 2 I ∈ R 4096×dx are image transformation matrices, b 1 I ∈ R 4096 and b 2 I ∈ R dx are bias vectors, and d x is the source words vector space dimensionality, all trained with the model.