At the encoding stage, we use the hierarchical bi-directional RNN to encode the sentences and the text document, use the RNN and the CNN to encode the image set.
a logistic classifier is then applied using features computed based on the RNN model.