Using hard negative examples has proven to be effective in the object detection to train a discriminative detector (Felzenszwalb et al., 2010;Shrivastava et al., 2016).
adding negative examples is usually not easy in the openvocabulary setting, because it is not guaranteed that a region without a positive label is negative.