As Table 3 shows, the data sources vary in coverage: it would be unfair to compare data sources on the different subsets of the evaluation datasets that they have coverage for.
when comparing data sources we want to make sure we evaluate on images for the exact same word pairs.