This problem is very much re-  lated to the object recognition task in computer vision, but instead of using just visual data and labels, these cross-modal models also utilize textual information (Socher et al., 2014;Frome et al., 2013).
nearest neighbors for the linguistic model tend to be of a more abstract nature: where we find mouth and throat as auditory neighbors for language, the linguistic model gives us concepts like word and dictionary; while auditory gossip sounds like maids and is something you might do in the corridor, it is linguistically associated with more abstract concepts like news and newspaper.