The objective for the multi-modal skip-gram has an additional visual objective J vis (in this case a margin criterion): Here, we take a similar but more straightforward approach by making the auditory context a part of the initial training objective, which is possible because linguistic and auditory representations have the same dimensionality.
we modified word2vec to predict additional auditory contexts that have been set to the corresponding BoAW representation.