There are two strategies that we could use in deciding whether to align a scene graph node d (whose label space is O ∪ A ∪ R) with a word/phrase w in the sentence: • Word-by-word match (WBW): d ↔ w only when d's label and w match word-for-word.
we thank Peter Anderson, Sebastian Schuster, Ranjay Krishna, Tsung-Yi Lin for comments and help regarding the experiments.