Inferring the objects from the vision side helps learn the object relationships, and inferring from the language side helps learn the crossmodality alignments.
the learning rate is set to 1e−4 instead of 5e − 5.