To empirically motivate our generation models, as a preliminary experiment we investigate the level of coordination between visual attention and linguistic production in the DIDEC dataset.
we test whether scanpath similarity and sentence similarity are correlated and whether taking into account the sequential nature of the two modalities results in higher cross-modal alignment.