This is somewhat surprising, as a model which has access to image information at each time step should be less likely to "forget" image content and hallucinate objects.
it is possible that models which include image inputs at each time step with no access to spatial features overfit to the visual features.