1 Research in multi-modal semantics deals with the grounding problem (Harnad, 1990), motivated by evidence that many semantic concepts, irrespective of the actual language, are grounded in the perceptual system (Barsalou and Wiemer-Hastings, 2005).
recent studies have shown that performance on NLP tasks can be improved by joint modeling of text and vision, with multimodal and perceptually enhanced representation learning outperforming purely textual representa-tions (Feng and Lapata, 2010;Kiela and Bottou, 2014;Lazaridou et al., 2015).