There is also work using deep learning methods to project different modality inputs into a common space, including restricted Boltzman machines (Ngiam et al., 2011;Srivastava and Salakhutdinov, 2012), autoencoders (Silberer and Lapata, 2014;Silberer et al., 2016), and recursive neural networks (Socher et al., 2013).
the above methods can only generate multimodal vectors of those words that have perceptual information, thus reducing multimodal vocabulary drastically.