Simply put, language does not occur in a vacuum.
to prior work in machine learning, where mappings between vision and language have been examined (Kiros et al., 2014;Vinyals et al., 2015;Xu et al., 2015), our goal in integrating visual and linguistic data is not to accomplish a task such as image search/captioning that inherently requires a mapping between these modalities.