The inference network takes as input the features X and learns a corresponding hidden representation h. More specifically, it outputs two components: (µ, ς) corresponding to the mean and log-variance of the gaussian parametrizing the hidden representation h. We model this using a "multi-view" network which incorporates hidden representations learned from multiple modalities into a joint representation.
given ddimensional hidden representations corresponding to multiple modalities z title , z network , and z content the model first concatenates these representations into a single 3d-dimensional representation z concat which is then input through a 2-layer feed-forward network to output a d-dimensional mean vector µ and a d-dimensional log-variance vector ς that parameterizes the latent distribution governing h. We now discuss the models used for capturing each view.