Training with perceptual context improves multimodal neural models compared to training on language alone.
augmenting a predictive language model with images that illustrate the sentences being learned enhances its next-word or masked-word prediction ability.