To achieve this, we propose to reconstruct the vector of associated word from the corresponding multimodal semantic representation.
in the decoding process we change the equation 3to: and equation 4to: x t = g(W tĥ t + bt) x a = g(W aĥ a + bâ) To train the model, we add an additional objective function, which is the mean square error between the embeddings of the associated word y and their re-constructive embeddingsx asc : where y i and x i are the embeddings of a pair of associated words.