The experiments that evaluated models on individual frames (Section 4.3) from the video have trends similar to those seen on mean pooled frame features.
the model trained on Flickr30k, when directly evaluated on YouTube video frames performs better on subjects and verbs, whereas the one trained on COCO does better on objects.