We compare our JTAV framework with: • doc2vec (Le and Mikolov, 2014), a paragraph embedding approach; • lyricsHAN (Alexandros, 2017), a hierarchical attention network which encodes songs into vectors; • BiGRU, a bidirectional GRU network which utilizes the song lyrics as input; • attBiGRU, our novel text modeling approach which adds the available captions as the supplying role based on BiGRU; • MFCC (McFee et al., 2017), a type of cepstral representation of the audio clip generated by librosa; • convnet (Choi et al., 2017a), an acoustic feature learning method based on transfer learning; • DCRNN-MelS, our novel audio modeling approach which takes MelSs as input; • DCRNN-CQT, our novel audio modeling approach which takes CQTs as input; • F-VGG (Simonyan and Zisserman, 2015), a VGG network pre-trained on the ImageNet dataset and fine tuned on the MoodShutter dataset; • F-ResNet , a ResNet pre-trained on the ImageNet dataset and fine tuned on the MoodShutter dataset; • F-DenseNet , a DenseNet pre-trained on the ImageNet dataset and fine tuned on the MoodShutter dataset; and • early fusion, a method which combines features learned by our methods before classification or regression tasks, and is very popularly used (Chen et al., 2017;Oramas et al., 2017).
the first is bi-modal retrieval, which utilizes one modality to find similar content in the other modality, such as finding the most relevant texts to a given image .