This happens because such additional objects do not seem too relevant to humans, as these are mostly not mentioned in the references.
the scores from VIFIDEL change with both number of annotations and the reference descriptions, while metrics like METEOR (0.354) and SPICE (0.320) would remain constant.