Since it is possible that different verbs may be associated with varying number of object arguments, we limit ourselves to verbs that take on at most two objects (or more specifically two noun phrase arguments) as a simplifying assumption: N k = {n 1 , n 2 } where n 2 can be NULL.
n 1 and n 2 are the subject and direct objects associated with v ∈ V. Using this assumption, we can construct the set of verbs, V. To do this, we use human labeled descriptions of the training images from the UIUC Pascal-VOC dataset (sec.