Inferring the objects from the vision side helps learn the object relationships, and inferring from the language side helps learn the crossmodality alignments.
we perform two sub-tasks: RoI-Feature Regression regresses the object RoI feature f j with L2 loss, and Detected-Label Classification learns the labels of masked objects with cross-entropy loss.